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Overview 


Too many students in high-poverty urban communities drop out of high school, and too few gradu- 
ate prepared for college and careers. Three national organizations — Talent Development Second- 
ary, City Year, and Communities In Schools — partnered to form Diplomas Now in an effort to turn 
those numbers around. Supported by funds from a U.S. Department of Education 2010 Investing in 
Innovation (i3) validation grant and private sources, Diplomas Now teams have been implementing 
their data-driven, tiered intervention model in urban secondary schools across the nation. The model 
combines a comprehensive school reform strategy, intended to transform the academic experience of 
all students, with more targeted interventions for students who have “early warning indicators” relat- 
ed to attendance, behavior, and course performance. By identifying students at risk of dropping out 
and providing individual support, Diplomas Now attempts to get struggling students back on a stable 
trajectory toward their diplomas. 

MDRC and 1CF International are conducting an independent, experimental evaluation of the impact 
and implementation of Diplomas Now. Sixty-two secondaiy schools in 1 1 school districts agreed to 
participate in this study between 2011 and 2013. Thirty-two of these schools were randomly as- 
signed to implement the Diplomas Now model while the other 30 schools were assigned to a control 
group, continuing their existing school programs or implementing other reform strategies of their 
choosing. Two prior evaluation reports focused on the first two years of Diplomas Now implementa- 
tion. This third report shares interim impact findings for those years, paying particular attention to 
attendance, behavior, and course performance outcomes of students in sixth or ninth grade, their 
transition year into middle or high school — the first-year impacts of a multiyear program. 

• The Diplomas Now model produced a positive and statistically significant impact on the per- 
centage of students with no early warning indicators — students with better than 85 percent at- 
tendance, fewer than three days suspended or expelled, and passing grades in both Eng- 
lish/language arts and math. Helping students maintain or reach these thresholds is an explicit 
target of Diplomas Now school teams. 

• Diplomas Now did not have a statistically significant impact on the percentage of students meet- 
ing a more stringent threshold suggestive of a more stable educational trajectory: better than 90 
percent attendance, no suspensions or expulsions, and passing all four core subject areas of Eng- 
lish/language arts, math, social studies, and science. 

• Diplomas Now did not produce a significant impact on average attendance, discipline, and 
course passing rates in sixth and ninth grades compared with rates at schools that did not im- 
plement the model. These outcomes improved from baseline in both Diplomas Now and com- 
parison schools. 

• There were more promising impacts for middle schools than for high schools. In middle 
schools, Diplomas Now had a positive, statistically significant impact on the percentage of 
sixth-graders with no early warning indicators. There were no significant impacts, positive or 
negative, on the attendance, behavior, and course performance outcomes of ninth-graders. 

• Students at Diplomas Now schools reported participating in more academically focused after- 
school activities, and more reported having a positive relationship with an adult at school who is 
not a teacher, than their peers in the comparison schools. Students in both groups of schools re- 
ported similar perceptions of school safety and climate, and the Diplomas Now model did not 
have an effect on students’ self-perceptions or school behaviors. 
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Preface 


American education has entered its next phase with the signing of the Every Student Succeeds 
Act (ESSA) by President Obama in December 2015. In a change from the prior education law 
(No Child Left Behind), states rather than the federal government now play the lead role in the 
identification of struggling schools and the kinds of interventions that can help them improve. 
Stressing accountability, ESSA encourages states to bring evidence-based practices to bear in 
this endeavor. The federal Investing in Innovation (i3) evaluation of Diplomas Now is a notable 
effort to collect such evidence. 

Diplomas Now is a secondary school reform model that aims to change school 
structures and practices in ways that will affect students’ engagement and persistence, starting 
with their transition year into middle school (sixth grade) or high school (ninth grade). The first 
two reports from this evaluation focused on the implementation of the Diplomas Now model. 
This third report provides a valuable interim look at the reform model’s impact on students’ 
attendance, behavior, and course performance — predictors of whether students graduate or 
drop out — as they begin their middle school or high school journeys. A later report will discuss 
the quality of model implementation across four years and its impact on two key four-year 
outcomes: ninth-grade completion for middle school students and high school graduation. 

The opportunity for rigorous study of a whole school reform, nationally scaled and 
implemented for four or more years, is rare, and the signing of ESSA only increases its 
relevance. The information generated from the full scope of this evaluation will contribute 
meaningfully to the existing knowledge base of how best to support underserved students and 
communities, and can help states and their local education agencies make more thoroughly 
informed decisions about how they can improve the secondary schools that are most in need. 

Gordon L. Berlin 
President 
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Executive Summary 


Although the national high school graduation rate has increased over the past decade, one in 
five students still do not complete high school in four years. 1 Among low-income students, al- 
most 30 percent fail to graduate on time. 2 Compared with high school graduates, dropouts are 
more likely to earn less money, live in poverty, suffer from poor health, be incarcerated, or be 
dependent on social services. 3 

Diplomas Now is a partnership of three national organizations — Talent Development 
Secondary, City Year, and Communities In Schools — collaborating in an effort to transform 
urban secondary schools so that fewer students drop out and more graduate ready for postsec- 
ondary education and work. The Diplomas Now model is a comprehensive multiyear approach 
to whole-school reform that includes structural change, instructional materials and curricula, 
teacher and administrator coaching and support, and an early warning indicator and intervention 
system to identify and support students falling off track for graduation. The program brings ad- 
ditional human resources into the school both to bolster implementation of the model and to 
provide direct assistance to students. With the goal of a continuous system of support through 
secondary school, the model seeks to help more students graduate by improving their attend- 
ance, behavior, and course perfomiance, particularly in English/language arts and math, during 
the middle grades and high school. 

Acting as a representative for the partnership, Johns Hopkins University, home to Tal- 
ent Development Secondary, was awarded an Investing in Innovation (i3) validation grant by 
the U.S. Department of Education in 2010 to support the expansion of Diplomas Now from a 
few schools to more than 30 middle and high schools in more than 10 school districts. The grant 
funds also support a rigorous random assignment evaluation of the Diplomas Now model, led 
by MDRC. 

This report discusses the early impacts of the Diplomas Now model on student and 
school outcomes at the end of the first and second years of model implementation. It focuses in 
particular on students during sixth and ninth grades, critical transition years into middle and 
high school. Accordingly, this report presents the first-year impacts of a multiyear program. 


'Richard J. Murnane, “U.S. High School Graduation Rates: Patterns and Explanations,” Journal of Eco- 
nomic Literature 51,2 (2013): 370-422. 

2 Robert Balfanz, John M. Bridgeland, Joanna Hornig Fox, Jennifer L. DePaoli, Erin S. Ingram, and Mary 
Maushard, Building a Grad Nation: Progress and Challenge in Ending the High School Dropout Epidemic 
(Washington, DC: Civic Enterprises, 2014). 

"Kristin Anderson Moore, Making the Grades: Assessing the Evidence for Integrated Student Supports 
(Bethesda, MD: Child Trends, 2014). 
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The Diplomas Now Model 

The Diplomas Now partnership works with schools to ensure that students are getting the sup- 
port they need to (1) get to school and to class, (2) behave in ways that facilitate learning, and 
(3) keep up with the lessons being taught. In other words, the pathway to student success in 
schools using the model is linked to attendance, behavior, and course performance: the “ABCs” 
that predict whether students graduate or drop out. 4 The Diplomas Now model is a multidimen- 
sional system of organizational and instructional reforms and targeted student support services 
organized under the following Four Pillars. 5 

Pillar I. Teacher Teams and Small Learning Communities 

Diplomas Now collaborates with school leaders to reorganize schools so that small 
groups of teachers work consistently with the same population of students. Not only does this 
allow teams of teachers to work together, the better to teach and support their students; it also 
creates a sense of community among the students. These teacher teams and small learning 
communities function best when teachers have a chance to collaborate within the daily schedule 
and when classes are long enough to cover material in depth and keep up the pace of instruction. 

Pillar II. Curriculum and Instruction with Professional Development 

This pillar is focused on teaching and learning, and on giving teachers the training and 
resources they need to deliver strong lessons. Through professional development that includes 
an intensive peer coaching system for math and English/language arts teachers, teachers have an 
opportunity to sharpen their pedagogy. Diplomas Now also offers curricular materials aligned 
with college- and career-ready standards and ensures that schools offer accelerated remediation 
courses for struggling students so that all students can meet their potential. 

Pillar III. Tiered Student Supports 

The Diplomas Now partners collaborate to help schools provide the right services to the 
right students at the right time and at the right level of intensity. 6 To do so, they use early wam- 


4 Robert Balfanz, Liza Herzog, and Douglas J. Mac Iver, “Preventing Student Disengagement and Keeping 
Students on the Graduation Path in Urban Middle-Grade Schools: Early Identification and Effective Interven- 
tions,” Educational Psychologist 42, 4 (2007), 223-235. 

5 For more information on the Diplomas Now model, see William Corrin, Susan Sepanik, Aracelis Gray, 
Felix Fernandez, Ashley Briggs, and Kathleen K. Wang, Laying Tracks to Graduation: The First Year of Im- 
plementing Diplomas Now (New York: MDRC, 2014). 

6 “Tiered Student Supports” refers to different levels of support offered across the school or to individual 
students based on need. Tier I interventions support the entire school. Tier II interventions are individual sup- 
port services offered to students identified as falling off track. Many of these interventions are provided by City 
Year AmeriCorps members who serve as mentors, tutors, and role models. Tier III interventions are for stu- 
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ing indicators to identify students who require different types of support. Teachers, administra- 
tors, and Diplomas Now staff members meet regularly to review students’ attendance rates, dis- 
ciplinary referrals or suspensions, and course performance and to plan interventions for students 
in need of support. 

Pillar IV. Can-Do Culture and Climate 

School reform is difficult, and school staff members often have much to do when they 
are asked to effect change. Diplomas Now brings at least a dozen staff members to a school to 
help coordinate the transformation, introduce new practices and structures, provide training and 
support to school staff members, provide additional services to students, and engage with fami- 
lies and community organizations. Providing resources to assist the school’s staff helps foster a 
culture and climate in which it feels possible to improve the school. 


The National i3 Evaluation of Diplomas Now 

In total, 62 schools (33 middle schools and 29 high schools) from 1 1 large urban school districts 
across the country were recruited to participate in the study starting in either the 2011-2012 or 
the 2012-2013 school year. 7 By design, Diplomas Now works in high-needs schools. The par- 
ticipating schools, all eligible for Title I funds, 8 serve large populations of low-income and mi- 
nority students. Thirty-two of the participating secondary schools were randomly assigned to 
implement the Diplomas Now model (DN schools), and 30 were assigned to continue with 
“business as usual” (non-DN schools), either maintaining their existing practices and structures 
or pursuing other types of school reform. 9 This random assignment design is often referred to 
as the “gold standard” in evaluation because the schools are all similar at the beginning of the 
study, and the decision about which schools will implement the program is not related to any 
preexisting characteristics of the schools. Therefore, any differences between the DN and non- 
DN schools that emerge after random assignment can be attributed to the program rather than to 
school characteristics; that is, Diplomas Now caused the observed differences. 


dents at the highest risk of dropping out and are generally coordinated by a case manager from Communities In 
Schools. 

7 Five of the school districts are among the 20 largest in the country, and all but 1 are among the 100 larg- 
est. Chris Plotts and Jennifer Sable, Characteristics of the 100 Largest Public Elementary’ and Secondary 
School Districts in the United States: 2007-08, NCES 2010-349 (Washington, DC: National Center for Educa- 
tion Statistics, U.S. Department of Education, 2010). 

8 Title I funds from the U.S. Department of Education go to schools with high numbers or high percentages 
of students from low-income families. 

9 Two middle schools are not included in the analyses in either the first or second year due to issues with 
grade configuration. Two more middle schools are not included in the second-year analyses because one 
school closed and one school stopped serving sixth grade. 
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The study’s experimental design makes it possible for the evaluation to assess the im- 
pact of Diplomas Now. This third report focuses on the early impacts of Diplomas Now on 
students’ attendance, behavior, and course perfonnance measures (the ABC outcomes), sepa- 
rately and in combination, during their first year in middle school or high school over the 
course of the first two years that the model was implemented in participating schools . 10 Does 
the implementation of Diplomas Now have an impact on how many students are on a path to 
high school graduation by the end of their first year of middle school or high school? During 
that first year, what difference does Diplomas Now make for attendance rates, suspensions and 
expulsions, and successful course completion? This report also discusses the impact of Di- 
plomas Now on possible precursors to the ABC outcomes, such as the climate of the 
school, support from parents and the community, and students’ attitudes and relationships. 


Early School and Student Outcomes 

The Diplomas Now model is hypothesized to achieve its intermediate goals of improving at- 
tendance, behavior, and course perfonnance, especially in English/language arts and math, 
through several mediating pathways. These early precursor outcomes include measures of posi- 
tive school climate, the addition of academic after-school activities, and increased parent and 
community support, along with measures of student attitudes and behaviors, including self- 
confidence, engagement and effort in school, study habits, and relationships with adults and 
peers. Administrator, teacher, and student survey items were used to measure these outcomes. 
Analyses of Diplomas Now’s impact on these outcomes found the following: 

• The Diplomas Now model had positive and statistically significant impacts 
on teachers’ perceptions of school climate during the second year of imple- 
mentation. There were no other statistically significant impacts on early 
school outcomes as reported by administrators and teachers, but the findings 
tend to point in a positive direction. 

• Students at DN schools reported participating more in academically focused 
after-school activities than their peers at non-DN schools. Students at both 
groups of schools reported similar perceptions of safety, climate, and behav- 
ioral issues at school during the first year of implementation . 1 1 


10 The primary analyses for this report focus on the second year of implementation, when the model was 
more mature. 

11 Sixth- and ninth-grade students were surveyed in the spring of the first year of implementation, so all 
early school and student outcomes coming from student survey items represent the first rather than the second 
year of implementation. 
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• The Diplomas Now model did not have an effect on students’ self- 
perceptions and school behaviors as measured by the student survey during 
the first year of implementation. 

• Students at DN schools were more likely to report a positive relationship 
with an adult at school who was not a teacher, but the Diplomas Now model 
had no impact on student perceptions of relationships with teachers, adminis- 
trators, and other students. 

The areas in which Diplomas Now is having some positive effects — teachers’ percep- 
tions of the climate of the school, and students’ participation in after-school activities and 
relationships with adults at school — align with specific refomis implemented as part of the 
Diplomas Now model. 


ABC Outcomes 

The long-term goal of the Diplomas Now model is to increase student graduation rates and col- 
lege readiness by improving students’ success on the indicators that past research has suggested 
are connected to graduation: the ABC outcomes of attendance, behavior, and course perfor- 
mance. 12 Thus far, the study team has been able to explore early impacts for sixth- and ninth- 
grade students making the transition into Diplomas Now secondary schools during the first two 
years of model implementation in those schools. The reported impacts of the model on student 
outcomes are based on the first year of students’ experience with this multiyear intervention. 

The Diplomas Now model includes structures and practices intended to help students 
stay above or move above two ABC outcome thresholds: The first serves as an intervention 
standard, below which the model targets students for additional support, and the second indi- 
cates a more secure course to graduation. Specifically, the first identifies students with “early 
warning indicators” of being at risk of not progressing successfully to high school graduation: 
daily attendance of 85 percent or less, suspensions or expulsions for a total of three or more 
days, and failing grades in English/language arts or math classes. Diplomas Now staff members 
work with school staff members in DN schools in an effort to increase the number of students 
without early warning indicators. Over time, Diplomas Now implementation ideally will in- 
crease the number of students who meet the second threshold, indicating a more stable educa- 
tional trajectory: having better than 90 percent daily attendance, no suspensions or expulsions, 


12 Elaine Marie Allensworth and John Q. Easton, The On-Track Indicator as a Predictor of High School 
Graduation (Chicago: University of Chicago Consortium on Chicago School Research, 2005); Balfanz et al. 
(2007); James J. Kemple, Micha D. Segeritz, and Nickisha Stephenson, “Building On-Track Indicators for 
High School Graduation and College Readiness: Evidence from New York City,” Journal of Education for 
Students Placed at Risk (JESPAR) 18, 1 (2013): 7-28. 
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and passing grades in all core courses (that is, English/language arts, math, social studies, and 
science). These higher “stability” thresholds represent normative expectations of secondary 
school students: to go to school regularly, to stay out of serious trouble, and to pass their classes. 
For students at this level, whole-school programming and instruction are deemed adequate sup- 
port. Students in between the two levels are monitored by Diplomas Now and school staff 
members and may receive additional attention as needed. 

Impacts for the Full Sample of Schools 

The evaluation team analyzed whether implementation of Diplomas Now had an im- 
pact on the percentages of students at both stability and early warning thresholds for each of the 
ABC outcomes, as well as the percentages of students at these threshold levels for all three of 
the outcomes combined. The results of these analyses are summarized in Figure ES.l. 

• For the combined sample of sixth- and ninth-graders in the second year of 
implementation (that is, the second cohort of students), there was a statistical- 
ly significant 3.6 percentage point impact of Diplomas Now on the percent- 
age of students with no early warning indicators. That is, DN schools were 
more successful than non-DN schools in helping students stay above or move 
above the early warning thresholds for all three ABC outcomes in combina- 
tion. The Diplomas Now model did not produce a statistically significant im- 
pact on the percentage of students meeting the stability threshold across all 
three ABC outcomes. (See the ABC composite section of Figure ES.l.) 

• The Diplomas Now model did not produce a statistically significant impact 
on the percentage of students above either the stability or early warning 
thresholds for any of the separate ABC outcome measures. 13 (See the attend- 
ance, behavior, and course perfonnance sections of Figure ES.l.) 

Table ES.l displays the impacts of the implementation of the Diplomas Now model on continu- 
ous measures of attendance, behavior, and course performance outcomes — for example, the 
average percentage of days attended by students rather than the percentage of students meeting 
an attendance threshold — for the combined sample of sixth- and ninth-grade students. These 
analyses provide insight into whether the implementation of the Diplomas Now model, with its 
targeted interventions for struggling students and broader instructional and structural reforms, 
had an impact, on average, across the sixth and ninth grades. 


1 3 A student with only one early warning indicator will fall below the composite threshold; therefore, al- 
though impacts may not be significant when outcomes are measured separately, it is possible that these impacts 
in combination translate to lifting enough students over the composite threshold to be significant cumulatively. 
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Figure ES.l 

Percentage of Students At or Above Threshold Measures, 
by DN and Non-DN Schools, Cohort 2 
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SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

“Behavior outcomes include suspensions and expulsions. 

b Measure indicates the percentage of students who passed all core courses (math. English/language arts, science, 
and social studies courses), all core math courses, and all core English/language arts courses, respectively. 

“Students with stability indicators attended over 90 percent of days enrolled in the district, were never suspended or 
expelled, and did not fail any core courses (math, English/language arts, science, or social studies) attempted during 
the school year. 

d Students with no early warning indicators attended over 85 percent of the days enrolled in the district, were 
suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during the 
school year. 
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Table ES.l 


Impacts on Continuous Measures of 
Attendance, Behavior, and Course Performance, Full Sample, Cohort 2 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P-Value for 

Estimated 

Impact 

Attendance 

Percentage of enrolled days attended 

89.7 

89.3 

0.4 

0.03 

0.602 

Behavior 

Percentage of enrolled days suspended or expelled 

1.5 

1.2 

0.3 

0.05 

0.177 

Course performance 

Percentage of core courses passed 2 

86.6 

86.6 

0.1 

0.00 

0.924 

Sample size 

29 

29 





SOURCE: MDRC calculations based on student records obtained from school districts. 


NOTES: Across 58 study schools, 14,950 nonrepeating sixth- and ninth-grade students are included in the 
analyses. Among the sample, 6,997 students attended Diplomas Now (DN) schools and 7,953 students 
attended non-DN schools. Some students are not included in the analyses of course performance measures 
because data were not available on their grades for specific courses, and an entire DN middle school was 
dropped from these analyses because there were no baseline course data for the students attending that school. 
There are no more than 8 percent missing DN school students and 8 percent missing non-DN school students 
for any of the course performance measures. 

Estimated impacts are based on a two-level model with students nested within schools controlling for 
random assignment block and school- and student-level covariates. The “DN Schools” and “Non-DN Schools” 
columns display regression-adjusted mean outcomes for each group, using the mean covariate values for 
students in the “DN Schools” column as the basis for the adjustment. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

'The denominator includes all core courses (math. English/language arts, science, or social studies) each 
student attempted during the school year. 


• Diplomas Now did not produce a statistically significant impact on students’ 
attendance rates, percentage of days suspended or expelled, or percentage of 
core courses passed. 

Thus, after two years of model implementation, the Diplomas Now model has neither increased 
average attendance and course passing rates nor decreased disciplinary days beyond the levels 
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achieved in the non-DN schools; both groups of schools experienced similar levels of im- 
provement on these measures. 

Impacts for Middle Schools and High Schools 

Middle schools and high schools typically vary in size and structure and serve students 
at different places in their developmental and educational pathways. Furthermore, ninth-graders 
have had more time to accumulate early warning indicators through middle school and may 
reach the ninth grade further off track. 14 Given these differences, some aspects of the Diplomas 
Now model vary for middle schools and high schools. Therefore, impacts on student outcomes 
were analyzed for these two groups of schools separately as well as together. 

• The Diplomas Now model had a positive and statistically significant impact 
on the percentage of students with no early warning indicators in middle 
school. (See Figure ES.2.) 

• The Diplomas Now model had a positive and statistically significant impact 
on the percentage of middle school students who attended over 90 percent of 
enrolled days. 

Although there are no other statistically significant impacts for middle schools, the middle 
school impact estimates across multiple analyses were more often positive than were those for 
high schools. 

• On average, Diplomas Now did not produce statistically significant impacts 
at the high school level compared with the outcomes at non-DN schools. 

Overall, results of the evaluation analyses suggest that Diplomas Now may have had a 
more positive early impact in middle schools than in high schools. While DN high schools did 
see improvements across ninth-grade attendance, behavior, course performance, and composite 
measures from the first to the second year of implementation, there were similar gains in the 
non-DN high schools as well. 

Impacts for More- and Less-Prepared Students 

The Diplomas Now model seeks to help struggling students overcome early warning 
indicators and get on a more stable pathway to graduation as well as to prevent students who are 
on a more stable educational trajectory from slipping off that path. Thus, the evaluation team 
analyzed the impact of Diplomas Now on the outcomes of students entering high school or 


14 Ruth Curran Neild, Robert Balfanz, and Liza Herzog, “An Early Warning System,” Educational Lead- 
ership 65, 2 (2007): 28-33. 
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Figure ES.2 


Diplomas Now Impacts on ABC Composite Measures, 
Middle and High Schools, Cohort 2 



Stability indicators 3 No early warning indicators' 3 


□ Middle school ■ High school 

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a Students with stability indicators attended over 90 percent of days enrolled in the district, were never suspended 
or expelled, and did not fail any core courses (math. English/language arts, science, or social studies) attempted 
during the school year. 

b Students with no early warning indicators attended over 85 percent of the days enrolled in the district, were 
suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 
the school year. 


middle school according to how prepared they were for the transition, based on whether enter- 
ing high school students were above or below the stability threshold at the end of eighth grade, 
and on middle school students’ levels of academic proficiency at the end of fifth grade. 15 

• Although Diplomas Now had no statistically significant positive impacts at 
the high school level on average, implementation of the model had more suc- 
cess keeping the stable students above the stability threshold than moving the 
less stable students above that threshold. 


15 Because elementary school data across participating study districts did not consistently include the disci- 
plinary and course performance metrics used to create the composite ABC indicators, the evaluation team used 
standardized state assessment scores to represent students’ preparedness for middle school. 
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• The Diplomas Now model did have a statistically significant positive impact 
on the percentage of more-prepared students who passed math in ninth grade. 

The students entering high school below the stability threshold — those who had been 
absent at least 10 percent of the time, been suspended or expelled, or failed a core course in 
eighth grade — posed a challenge for both the DN and non-DN schools. For example, in both 
groups of schools only about 30 percent of less-prepared students had no early warning indica- 
tors in ninth grade (compared with more than 70 percent of the more-prepared students), and 
only about 20 percent were above the stability threshold (compared with about 60 percent of the 
more-prepared students). This finding reinforces how valuable intervening successfully with at- 
risk students during the middle grades might be. 

• In general, the patterns of impacts were similar for sixth-grade students 
whether or not they were proficient on state English/language arts and math 
assessments at the end of elementary school. 


Next Steps 

After two years, implementation of the Diplomas Now secondary school reform model pro- 
duced a statistically significant, positive impact on the percentage of students with no early 
warning indicators, suggesting that a lower percentage of students were notably off track on the 
pathway to high school graduation at DN schools compared with non-DN schools. The focus of 
the Diplomas Now model, particularly with its early warning system and tiered support, is to 
reduce the percentage of students with early warning indicators, and this finding suggests those 
efforts are starting to make a difference after two years. The model’s effect on the percentage of 
students meeting the “higher bar” indicating a stable pathway to graduation is not statistically 
significant, but it shows progress: The 2.5 percentage point impact for the second cohort of 
sixth- and ninth-grade students was about 5 percentage points higher than the impact for the first 
cohort of students (-2.6 percentage points). 16 The increased impact for the second cohort aligns 
with two-year implementation findings from this evaluation. From the first to the second year of 
implementation, DN schools became more different from non-DN schools in terms of their use 
of refomi-oriented practices and structures — DN schools showed sustained levels of reform 
implementation, while the levels declined in the non-DN schools. 17 


16 The impact on the percentage of students with no early warning indicators increased 3.4 percentage 
points from the first cohort to the second cohort. This increase was not statistically significant. 

17 Susan Sepanik, William Corrin, David Roy, Aracelis Gray, Felix Fernandez, Ashley Briggs, and Kath- 
leen K. Wang, Moving Down the Track: Changing School Practices During the Second Year of Diplomas Now 
(New York: MDRC, 2015). 
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Overall, the early impacts of the model during the second year of implementation were 
stronger for sixth-graders than for ninth-graders. In particular, higher percentages of sixth- 
graders in DN schools had better than 90 percent attendance and no early warning indicators 
than their peers in non-DN schools. Although ninth-graders in DN high schools did not have 
better outcomes compared with their peers at non-DN high schools, ninth-grade outcomes im- 
proved from the first to the second year of implementation at both groups of schools. 

Johns Hopkins University was granted funding from the Office of Innovation and Im- 
provement of the U.S. Department of Education to support an extension of this evaluation that 
will make it possible to analyze the impact of Diplomas Now on the longer-term student out- 
comes that represent the model’s primary target: What is the impact of Diplomas Now on high 
school graduation rates and on the ninth-grade success of students from Diplomas Now middle 
schools? This will allow the study team to see whether the increase in impacts from the first to 
the second year continues over a longer period of implementation, and how well implementa- 
tion is maintained over that period. Furthermore, given how much better the ninth-grade out- 
comes were for students entering high school above the stability threshold, the promising im- 
pact findings for middle school students are worth continued attention. If these impacts are 
maintained, following these students into high school will indicate whether Diplomas Now’s 
encouraging intervention with students in the middle grades yields positive high school out- 
comes. The study team will also be able to explore the ABC outcomes, including the composite 
measures, of sixth- and ninth-graders in the fourth year of implementation, when the model has 
had even more time to mature in the schools. 

As of the publication of this report, the Diplomas Now partners had managed to main- 
tain model implementation for at least four years in almost all the DN schools. Since whole- 
school reform efforts implemented for several years have been shown to have greater impacts 
on student outcomes than those implemented for only a couple of years, analyzing the longer- 
term effects of this model is important. 18 Furthermore, the extension will also allow for some 
exploration of the variation in implementation and outcomes found across schools and students, 
which may support a better understanding of best practices and generate hypotheses that could 
inform practice moving forward. 


ls Geoffrey D. Borman, Gina M. Hewes, Laura T. Overman, and Shelly Brown, “Comprehensive School 
Reform and Achievement: A Meta-Analysis,” Review of Educational Research 73, 2 (2003): 125-230. 
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Chapter 1 

Introduction 


Background 

Although the national high school graduation rate has increased over the past decade, one in 
five students still do not complete high school in four years. 1 Among low-income students, al- 
most 30 percent fail to graduate on time. 2 Compared with high school graduates, dropouts are 
more likely to live in poverty, earn less money, suffer from poor health, be incarcerated, or be 
dependent on social services. 3 Students who face the most serious barriers to earning their di- 
plomas are in the greatest need of intensive academic, social, and other interventions to make it 
through high school, and most dropouts are concentrated in urban high schools in low-income 
communities. 

Research has shown that it is possible to predict a student’s likelihood of dropping out 
of high school using indicators of poor attendance, poor behavior, and course failure in mathe- 
matics and English/language arts measured as early as middle school. 4 These findings suggest 
that programs may have greater success getting more students to graduation if they intervene 
with students who are off track as early as middle school. Moreover, ninth grade is a critical 
year, and researchers have shown that improving student course performance in the ninth grade 
can lead to substantial improvements in graduation rates. 5 

Among those students who do graduate from high school, many do not graduate ready 
for college and need to take remedial (developmental education) courses: Over 30 percent of 
college undergraduates and about 40 percent of community college students enroll in such 
courses. 6 Beyond supporting struggling students to graduation, school improvement efforts need 
to ensure that all students participate and succeed in rigorous curricula that prepare them for 
college and careers. 


The Diplomas Now Partnership 

The Diplomas Now partnership — created by three national organizations, Talent Development 
Secondary, City Year, and Communities In Schools — works with schools to ensure that stu- 


'Murnane (2013). 

2 Balfanz et al. (2014). 

3 Moore (2014). 

4 Balfanz, Herzog, and Mac Iver (2007). 

5 Roderick, Kelley-Kemple, Johnson, and Beechum (2014). 
6 National Center for Education Statistics (2014). 
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dents are getting the support they need to (1) get to school and to class, (2) behave in ways that 
facilitate learning, and (3) keep up with the lessons being taught. In other words, the pathway to 
student success in Diplomas Now (DN) schools is linked to attendance, behavior, and coarse 
performance in mathematics and English/language arts: the “ABCs” that predict whether stu- 
dents graduate or drop out. The initiative targets underfunded urban secondary schools with 
many students who are not performing well academically, in communities struggling with pov- 
erty. The Diplomas Now partners collaborate to help schools provide the right services to the 
right students at the right time and at the right level of intensity. They do so by offering varying 
levels of support for students with different needs: whole-school restructuring and instructional 
reform to strengthen the educational experience of all students, individual support for students 
showing early signs of falling off track, and case management for students in need of deeper 
interventions. To determine which students need extra support, the model relies on regular mon- 
itoring of students’ attendance, behavior, and course performance in mathematics and Eng- 
lish/language arts. See Box 1.1 for more on what each of the Diplomas Now partners contrib- 
utes to the overall model. 7 

The Diplomas Now Model 

The Diplomas Now model is a multiyear, multidimensional system of organizational and in- 
structional reforms and targeted student support services. The elements of the model are classi- 
fied as nine “inputs,” some of which represent substantial interventions on their own, such as 
implementing a rigorous curriculum or setting up a tiered intervention system to identify at-risk 
students and tailor interventions to their specific needs. Diplomas Now integrates these inter- 
ventions into a cohesive model focused on ensuring that all students have a clear and achievable 
path to graduation. Eight of these inputs are implemented in collaboration with school staff 
members and align with the Four Pillars of Diplomas Now, a characterization of the model used 
by Diplomas Now staff members to help them organize their work. The Four Pillars and their 
associated inputs are presented in Figure 1.1. The ninth input (shown on the left-hand side of 
Figure 1.1) consists of professional development for the staff members of the Diplomas Now 
organizations, intended to give them the knowledge and skills they need to help schools imple- 
ment the Four Pillars. 

Pillar I: Teacher Teams and Small Learning Communities 

The Diplomas Now partners collaborate with school leaders to reorganize schools so 
that small groups of teachers work consistently with the same population of students. These small 


7 For a more detailed description of the partners and their roles, see Corrin et al. (2014), pp. 3-4, and 
Sepanik et al. (2015), pp. 2-3. 
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Box 1.1 


The Diplomas Now Partners and Their Roles 
Talent Development Secondary 

Talent Development Secondary, based at Johns Hopkins University, provides organiza- 
tional, instructional, curricular, and data support to schools intended to help all students 
achieve at high levels and prevent them from falling off track. This school-wide effort in- 
cludes reorganizing students and teachers into small learning communities, providing pro- 
fessional development and coaching to strengthen teacher pedagogy, and supplying col- 
lege and career preparatory course content. Talent Development Secondary employs a 
school transformation facilitator who works with school leaders to develop a systemat- 
ic school transformation plan, creates and manages an Early Warning Indicator data tool, 
organizes frequent multidisciplinary teacher-team data-response meetings that use the tool 
to guide a multitiered student support process, and works closely with local or regional 
instructional facilitators to oversee instructional and curricular reforms. School-based 
instructional coaches in English/language arts and math support teachers’ delivery of 
course content. For many students, whole-school organizational and instructional reforms, 
referred to as Tier 1 interventions, are enough to keep them on track. However, for some 
students, the Early Warning Indicator data indicate that additional and more intensive ser- 
vices are necessary. City Year and Communities In Schools play leading roles in provid- 
ing these additional services. 

City Year 

City Year is an AmeriCorps program through which young adults ages 18 to 24 partici- 
pate in a year of national service. A team of about a dozen City Year AmeriCorps 
members is assigned to a school, increasing the number of adults in a building paying at- 
tention to students and working with them both in- and outside of classrooms. The team is 
led by a City Year program manager and team leaders (typically second-year Ameri- 
Corps members), and its members are trained to provide a variety of academic and behav- 
ioral interventions — referred to as Tier 11 support — intended to help students stay on 
track to graduate. These “near peers” (given their proximity in age to the students) serve 
as tutors, mentors, and role models, personalizing the school experience of the students. In 
addition, the AmeriCorps members provide after-school programs and help teachers by 
working with students during class time. 

Communities In Schools 

Through a school-based site coordinator, Communities In Schools, a national dropout- 
prevention organization, draws on school and community resources to organize services 

— referred to as Tier 111 support — intended to move the students at the highest risk of 
dropping out back on track to graduation. The site coordinator assesses the needs of a stu- 
dent, develops an individual case plan to address those needs, and connects the student to 
services aligned with the case plan. Examples include professional counseling on anger 
management for a student with behavioral issues or long-term tutoring with a subject-area 
expert for a student far behind in class. A site coordinator will also provide direct service 

— for example, leading student discussion groups on topics like conflict resolution or the 
transition to adulthood. 
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Figure 1.1 
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learning communities create opportunities for personalization where teams of teachers know the 
same students and can work together to best teach and support them. Students also share the 
same classes and become known to one another. These teacher teams and small learning com- 
munities function best when there are opportunities for teachers to collaborate within the daily 
schedule and when they have classes long enough to cover material in depth and keep up the 
pace of instruction. This means that Diplomas Now staff members work with schools to reor- 
ganize master schedules to accommodate block scheduling and specific teacher collaboration 
time . 8 


Pillar II: Curriculum and Instruction with Professional Development 

This pillar is focused on teaching and learning, and on giving teachers the training and 
resources they need to deliver strong lessons. Through professional development that includes 
an intensive peer coaching system for math and English/language arts teachers, teachers have an 
opportunity to sharpen their pedagogy. For students, Diplomas Now offers curricular materials 
aligned with college- and career-ready standards; it also supports the provision of accelerated 
remediation courses for struggling students, so that all students can meet their potential. 

Pillar III: Tiered Student Supports 

Providing more intensive support for students with greater needs is the core idea of this 
pillar. The tiered intervention model involves implementing an early warning system that draws 
on data on the ABC indicators to identify students who are either off track or at risk of going off 
track. Students may be identified by a low attendance rate or disciplinary referrals or suspen- 
sions, or because they are failing or at risk of failing their mathematics or English/language arts 
courses — or a combination of these factors. Teachers, administrators, and Diplomas Now staff 
members participate in regular meetings to review data on these indicators and plan interven- 
tions for these students. The intensity of the interventions depends on student need and the de- 
gree to which the student is offtrack . 9 


8 “Block scheduling” refers to a school scheduling model in which daily class periods are longer than in 
traditional models (for example, 80 to 90 minutes rather than 45 to 60 minutes). This means that fewer class 
periods are scheduled per day in block-scheduled schools (four extended class periods, for example) than in 
traditional schools, where a daily schedule with six or more class periods is more common. 

9 The model has three tiers: Tier I interventions support the entire school. Tier 11 interventions are individ- 
ually tailored supports offered to students identified as falling off track. Many of these interventions are pro- 
vided by City Year Ameri Corps members, who serve as mentors, tutors, and role models. Tier III interven- 
tions are for students at the highest risk of dropping out and are generally coordinated by a case manager 
from Communities In Schools. 
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Pillar IV: Can-Do Culture and Climate 

School reform is difficult, and school staff members often have much to do when they 
are asked to effect change. Diplomas Now brings at least a dozen staff members to a school to 
help coordinate school transformation, introduce new practices and structures, provide training 
and support, provide additional services to students, and engage with families and community 
organizations. All of these staff members are trained by Diplomas Now before and throughout 
the school year. Providing and organizing resources to assist the school’s staff helps foster a 
culture and climate where it feels possible to improve the school and support students better. 

Implementing these four pillars is hypothesized to affect a series of school outcomes 
(such as the school’s climate and communication among stakeholders) and student outcomes 
(like study habits and engagement with school), leading to improvements in students’ attend- 
ance, behavior, and course performance in mathematics and English/language arts, which 
should in turn lead to increased high school graduation rates. 


The National i3 Evaluation of Diplomas Now 

In total, 62 schools (33 middle schools and 29 high schools) from 1 1 large urban school districts 
across the country were recruited to participate in the study starting in either the 2011-2012 or 
the 2012-2013 school year. 10 By design, Diplomas Now works in high-needs schools. The 
schools in the study come from high-poverty urban areas where students struggle academically 
and drop out at high rates. The participating schools, all eligible for Title I funds, 11 serve large 
populations of low-income and minority students (81 percent eligible for free or reduced-price 
lunch; 83 percent black or Hispanic). Furthermore, the high schools participating in the study 
have weak promoting power (56 percent), suggesting that they struggle to move students from 
ninth through twelfth grade on time. 12 Thirty-two of the participating secondary schools were 
randomly assigned to implement the Diplomas Now model (DN schools) and 30 were assigned 
to continue with “business as usual” (non-DN schools), either maintaining their existing prac- 
tices and structures or pursuing other types of school reform. This random assignment design is 
often referred to as the “gold standard” in evaluation because the schools are all similar at the 
beginning of the study, and the decision about which schools will implement the program is 
random and not related to any preexisting characteristics of schools. The implication of this de- 
sign means that any differences between the DN and non-DN schools that emerge after random 

10 Five of the school districts are among the 20 largest in the country, and all but 1 are among the 100 
largest (Plotts and Sable, 2010). 

1 'Title 1 funds from the U.S. Department of Education go to schools with high numbers or high percent- 
ages of students from low-income families. 

12 Corrin et al. (2014). “Promoting power” is calculated as the ratio of twelfth-graders to ninth-graders 
three years earlier. 
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assignment can be attributed to the program rather than to specific school attributes; in short, 
Diplomas Now caused the observed differences. 

The study’s experimental design makes it possible for the evaluation to assess the im- 
pact of Diplomas Now. This third report of the evaluation focuses on the impact of Diplomas 
Now on students’ ABC outcomes after one year of Diplomas Now intervention in the sixth and 
ninth grades. 13 For students who attend a Diplomas Now school in sixth or ninth grade, what 
difference does the Diplomas Now model make on attendance rates, suspensions and expul- 
sions, and successful course completion? Essentially, does the implementation of Diplomas 
Now keep more students on track to high school graduation by the end of their middle school or 
high school transition years? 

This report will also explore the impact of Diplomas Now on outcomes that may act as 
precursors to the ABCs. As shown in Figure 1.1, the theory of change for the Diplomas Now 
model posits that the multiyear implementation of the Four Pillars will affect early school and 
student outcomes (displayed in the first and second boxes on the top right-hand side of the fig- 
ure). The early school outcomes (shown in the top box) focus on changes the Diplomas Now 
partners would expect to see throughout a school as the model is implemented. The second box 
includes changes to student attitudes and behaviors that would be expected to occur as the mod- 
el is implemented and begins to “take hold” at a school. There are two possible scenarios for 
change depicted in the logic model. First, implementation of school reforms could lead to 
changes in early school outcomes that will subsequently affect student attitudes and behaviors. 
Second, implementation of services that support individual students could lead directly to 
changes in early student outcomes. In both these cases, the theory of change implies that posi- 
tive growth in student attitudes and behaviors should lead to better student attendance, fewer 
issues with behavior, and fewer course failures, and ultimately bring more students to graduate 
with the skills needed to be successful after high school. 

Johns Hopkins University, as a representative of the Diplomas Now partnership, was 
granted funding from the Office of Innovation and Improvement of the U.S. Department of Ed- 
ucation to support an extension of the evaluation. This extension will allow for a future report 
exploring variation in implementation and outcomes across schools that may generate hypothe- 
ses that could inform best practices, and a final report that will make it possible to analyze the 
impact of Diplomas Now on student outcomes after four years, answering the question: What is 
the impact of Diplomas Now on high school graduation rates and on the ninth-grade success of 


13 Two middle schools are not included in the analyses in either the first or second year due to issues with 
grade configuration. Two more middle schools are not included in the second-year analyses because one 
school closed and one school stopped serving sixth grade. 
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students from Diplomas Now middle schools? As the ultimate focus of the Diplomas Now 
model, this will offer the most definitive test of whether the model met its outcome goals. 


Summary of Key Findings from the First Two Reports 

The evaluation has produced two prior reports on the implementation of the Diplomas Now 
model, documenting how this complex, multicomponent reform intended to transform second- 
ary schools is implemented by multiple partners. The implementation research explores what it 
takes to put the model into practice, what factors promote or hinder implementation, and the 
nature of the collaboration among multiple actors from the Diplomas Now organizations and 
the schools. The first report from the evaluation, released in 2014, focused on program start-up 
and first-year implementation fidelity, as well as exploring qualitative data collected in a subset 
of DN schools on the collaborative processes taking place. The second report, released in 2015, 
continued the story, including findings about fidelity in the second year of model implementa- 
tion. It also examined school structure and staff practice at DN and non-DN schools to see 
whether Diplomas Now had created differences (that is, a “service contrast”) between the two 
groups of schools. Finally, it presented analyses of qualitative data, again collected in a subset 
of DN schools, exploring the context in which implementation was occurring, the integration of 
the model at schools, and the importance of staff stability. The following provides a summary of 
prior report findings. 

Fidelity of Implementation 

To create a measure of fidelity of implementation, each of the Diplomas Now partner 
organizations detailed the components it felt were needed for full implementation of the model. 
Over 100 components were identified, and each component was measured on a 0-1 scale with 0 
equaling no or low implementation and 1 equaling high fidelity to the model as originally in- 
tended. 14 The overall average fidelity score for all the components across all the DN schools 
was 0.61 during the first year and 0.62 during the second year. These scores suggest that on av- 
erage, schools were implementing a majority of the components with high fidelity to the model 
each year, but they still had room for growth to reach ideal implementation. The similarity in 
scores suggests stability in program implementation over the two years. Primary findings re- 
garding fidelity of implementation include: 

• DN schools were most successful in obtaining, retaining, and training the 
auxiliary staff members needed to implement the model effectively (Pillar 


14 The Diplomas Now national team and individual school teams are constantly working to improve the 
model and revise it to better support specific school needs. For this reason, some digression from the original 
model is expected. 



IV); using data to identify at-risk students; and collaborating to plan and pro- 
vide individual and small-group interventions for those students (Pillar III). 

Implementation fidelity grew from Year 1 to Year 2 in staffing and providing training 
to the auxiliary Diplomas Now staff, and by Year 2, almost all schools had all the necessary 
staff members in place to implement the model. A large majority of schools were also success- 
ful during both years in establishing a data system to track students’ attendance, behavior, and 
course perfonnance in mathematics and English/language arts and ensuring collaboration 
among staff members to plan and provide interventions for students falling off track. City Year 
AmeriCorps members (see Box 1.1) were successful in providing extra and complementary 
services to students across all schools, but some schools were less successful in ensuring that 
enough City Year AmeriCorps members were available to meet the needs of all the students. 
Schools were also moderately successful in establishing small learning communities of students 
who shared the same classes and teachers (Pillar I), but many schools struggled to hold frequent 
meetings of these communities’ interdisciplinary teams of teachers. 

• DN schools were least successful at offering teachers the peer coaching 
needed to strengthen practice, implementing curricular additions to ensure 
college and career readiness for all students (Pillar II), and involving parents 
and community members in school activities and decisions (Pillar IV). 

In both the first and second years of implementation, it was hard for schools to meet 
implementation goals in some areas where it was necessary to change school policy or structure 
or to get the school’s staff on board. This may have been due, in part, to the short timeline avail- 
able for recruitment of a large number of schools, which did not allow for the ideal amount of 
communication with school leaders ahead of implementation. Most schools struggled to achieve 
consistent coaching for math and English/language arts teachers at the level desired by Diplo- 
mas Now, and most schools did not provide the prescribed academic foundations and accelerat- 
ed remediation courses for struggling students. Finally, on average, schools were not meeting 
the ideal levels of implementation for involving parents and community members. 

Service Contrast 

DN schools were more likely to implement the types of activities found under several 
of the Diplomas Now pillars than non-DN schools, suggesting that Diplomas Now is making 
DN schools different from non-DN schools. Unlike the fidelity of implementation score, there 
was quite a bit of growth from Year 1 to Year 2 in service contrast. Key findings include: 

• There was service contrast between DN and non-DN schools for the pillars 
that showed strong and moderate implementation, including Pillar I (Teacher 
Teams and Small Learning Communities), Pillar III (Tiered Student Sup- 
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ports), and the part of Pillar IV (Can-Do Culture and Climate) focused on 
hiring and retaining the staff needed to implement the model. 

Teachers at DN schools were more likely than teachers in non-DN schools to report 
teaching in extended class periods and collaborating with interdisciplinary teams that shared the 
same students; using data to identify at-risk students and meeting with other school staff mem- 
bers to plan interventions for them; and seeing that students receive academic, behavioral, or 
emotional support. 

• Even though the implementation of teacher professional development and 
coaching did not fully meet the model’s goals, the service-contrast findings 
suggest that teachers at DN schools received more coaching than teachers at 
non-DN schools. Yet teachers at DN schools and teachers at non-DN schools 
reported similar usage of college readiness curricula. 

Math and English/language arts teachers at DN schools reported receiving more coach- 
ing than teachers at non-DN schools. Both groups of teachers reported relatively high levels of 
adoption of academic reform curricula, incorporation of transitional support classes for strug- 
gling students, and use of student-centered and college and career readiness-focused strategies 
in the classroom. 

• The contrast between DN and non-DN schools increased from Year 1 to 
Year 2 in several areas, including collaboration of teachers within interdisci- 
plinary teams (Pillar I), professional development of teachers (Pillar II), and 
the use of data to identify struggling students (Pillar III). 

From Year 1 to Year 2 practices and structures aligned with the Diplomas Now model 
were maintained or slightly improved in DN schools, while similar structures and practices 
were less evident in non-DN schools in Year 1 and became even less evident in Year 2, suggest- 
ing that Diplomas Now helped stabilize resources and programs at the DN schools. 

Challenges and Emerging Successes in Implementation During 

the First Two Years 

Effective collaboration is at the heart of the Diplomas Now school reform model, which 
deploys staff members from Talent Development Secondary, City Year, and Communities In 
Schools to work in concert with each other and with school staff members to implement the 
tiered intervention model. As part of its research during the first year, the evaluation team inves- 
tigated the collaboration of the Diplomas Now and school staff members at a subset of DN 
schools. 
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• Across the Diplomas Now and school staff members, two aspects of collabo- 
ration were most often reported to be important for successful implementa- 
tion: investment and role clarity. 

Administrators and teachers are key stakeholders whose engagement in implementing 
the Diplomas Now model inputs is indispensable. In order to become actively engaged, it is im- 
portant that they understand the model through activities such as information sessions and meet- 
ing school staff members at other DN schools. Continual communication, including regular 
meetings and informal check-ins, helps build the trust and acceptance necessary for the collabo- 
rative work of model implementation. In addition, it is crucial that Diplomas Now school-based 
staff members establish their purposes and roles with teachers and administrators as well as 
among themselves. The Talent Development Secondary, City Year, and Communities In 
Schools staff members provided schools with the human resources necessary for the implemen- 
tation of the Diplomas Now model. However, increased clarity regarding the roles and respon- 
sibilities of these Diplomas Now staff members would have fiirther improved model implemen- 
tation in the first year. 

During the second year, the research team focused on examining how school context af- 
fected implementation, exploring both the successes and struggles of implementing the pro- 
gram. Findings include: 

• Various factors external to the program (such as school closures, principal 
turnover, and budget cuts) influenced the implementation of Diplomas Now. 

DN and non-DN schools were probably equally affected by these issues, but 
in some cases Diplomas Now was able to offer schools consistency and addi- 
tional support in the face of the challenges. 

• Program staff members can foster stronger Diplomas Now implementation at 
a school in two main ways: (1) aligning program goals with school priorities 
and (2) securing administrator and teacher support for the Diplomas Now 
model. Qualitative data about Year 2 implementation indicates that Diplomas 
Now staff members understood how to do those two things, and as a result 
were able to make Diplomas Now “part of the school.” 

• Despite gains from Year 1 to Year 2 in hiring and training program staff 
members, in some schools program staff turnover may have undermined im- 
plementation and increased the burdens on the remaining staff members. On 
the other hand, maintaining at least some consistency in Diplomas Now 
leadership probably preserved useful institutional knowledge about the suc- 
cesses and challenges of the first year of Diplomas Now implementation, and 
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the collaborative relationships among partner organizations allowed for some 
flexibility to support consistent implementation. 


Orientation to This Report 

The ultimate goal of the Diplomas Now model is to lead more students to high school gradua- 
tion and college and career readiness. This report explores the impact of the Diplomas Now 
model during the first and second years of implementation on early and intermediate school and 
student outcomes, which are theorized to lead to the future goals of successful ninth-grade com- 
pletion for middle school students and high school graduation for high school students. Chapter 
2 discusses the methods used in analyzing the data for this report. It describes the outcome 
measures analyzed and the sample of students included in the analyses. It also compares base- 
line characteristics of DN and non-DN schools at random assignment to show whether the two 
sets of schools were equivalent before program implementation. Chapter 3 describes the early 
school and student outcomes as measured through administrator, teacher, and student survey 
data. Chapter 4 first discusses the impact of Diplomas Now on student ABC outcomes (attend- 
ance, behavior, and course performance) in the second year of implementation for all incoming 
sixth- and ninth-grade students. It also describes the findings for middle and high school stu- 
dents separately and looks at the impacts for subgroups of students based on their prior (that is, 
baseline) academic achievement and behavior. It then further explores the effectiveness of the 
model by reviewing impact findings for the sixth- and ninth-grade students during the first year 
of implementation and looking at the variation in impacts across schools in the study. 
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Chapter 2 

Study Design and Characteristics of Study Schools 


This chapter describes the design of the study, including the random assignment of schools; the 
research questions being answered and outcome measures reported on; and the samples of stu- 
dents, teachers, and administrators included in the analyses. Also discussed are the characteris- 
tics of schools and students in the study, along with the comparability of the schools implement- 
ing the Diplomas Now model (DN schools) and the schools not implementing the model (non- 
DN schools) at the time of random assigmnent. A brief description of how the results were ob- 
tained is also included, along with information for understanding impact findings. 

Key points from this chapter are as follows: 

• Random assigmnent resulted in two groups of schools (DN and non-DN 
schools) that were comparable on measures of attendance, student behavior, 
and course perfomiance at the start of the Diplomas Now model implementa- 
tion. In general, the students in the analytic sample who attended DN schools 
were similar to the students who attended non-DN schools. Baseline tests of 
characteristics of students in both sets of schools revealed no systematic dif- 
ferences, as expected in a randomized design. 

• The comparability of (1) the DN and non-DN schools at the start of the study 
and (2) the students entering DN schools and those entering non-DN schools 
bolsters confidence that the random assigmnent of schools to implement Di- 
plomas Now or not resulted in two equivalent groups, allowing for any dif- 
ference in outcomes between the DN and non-DN schools to be attributed to 
the implementation of the Diplomas Now model. 


The Random Assignment Design 

The core of the national evaluation of Diplomas Now is a school-level random assigmnent 
study design, presented in Chapter 1, in which some schools were randomly assigned to imple- 
ment the program (the Diplomas Now school reform model), and others were randomly as- 
signed not to implement the program but instead to continue with “business as usual” program- 
ming or other refonns. It is important to note that “business as usual” does not mean that these 
schools experienced no other refonns or interventions, but rather that they experienced whatev- 
er else a given school or district may have invested in during the study period, which was not 
Diplomas Now. This kind of study is considered the “gold standard” for evaluation because the 
random process creates conditions under which any differences in outcomes between irnple- 
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meriting and nonimplementing schools that are found after a program has been put in place can 
be causally attributed to that program as implemented. In other words, this method allows re- 
searchers to know with certainty that the program caused specific outcomes, and that impacts 
are not related to other school or student characteristics. 

The Diplomas Now evaluation recruited 62 schools to participate; 32 received the Di- 
plomas Now program, and 30 represent the “business as usual” comparison condition. Schools 
were recruited from 1 1 urban districts around the country. Within each district, schools were 
clustered into middle school or high school “blocks” and randomly assigned to implement Di- 
plomas Now (DN schools) or not (non-DN schools) within those blocks. In addition, because 
schools were recruited in two waves, they were also randomized by year. Overall, the block de- 
sign increases precision by ensuring that schools that are alike are compared with each other, 
and that similar numbers of middle and high schools within districts are randomized into each 
condition. Furthennore, assigning blocks that contain only schools recruited at the same time 
ensures that any differences in the temporal nature of recruitment and implementation do not 
influence outcomes. However, not all of the recruited schools are included in the main impact 
analyses. Two schools were dropped from the analysis because one closed after the first year of 
implementation and one stopped serving sixth grade, and the main analyses are focused on the 
impacts of Diplomas Now in the second year of implementation. Also, one middle school block 
was dropped because the DN school and the non-DN school in that block had different grade 
configurations, in which students in one school started in sixth grade and those in the other 
started in seventh grade. 1 Since these analyses compare students entering middle school (or high 
school) at the same grade level, students could not be compared across these schools. Thus, 
there are a total of 29 high schools (15 DN and 14 non-DN) and 29 middle schools (14 DN and 
15 non-DN) across 21 random assignment blocks in the analysis sample. 

Analyses for this report focus on the transition year for students (that is, the year stu- 
dents first enter middle school or high school) because the Diplomas Now model is designed to 
provide extra support to students at critical transition points, where they tend to falter and need 
additional support. Although findings are presented for two cohorts of students (that is, students 
who enter and travel through a grade together), the main analysis focuses on the second cohort 
of students to enroll in study schools after implementation, because schools may need at least a 
year to fully understand and adapt to all the new program elements. 


'This difference in grade configurations was discovered after random assignment had been conducted, 
but it was a preexisting situation and not related to assignment of the schools to implement DN or proceed 
with “business as usual.” 
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Characteristics of Schools and Students in the Study 

By design, the Diplomas Now program aims to work with schools facing many challenges, typ- 
ically schools whose students underperform academically and are at risk of eventually dropping 
out. These are the kinds of schools participating in this evaluation. In addition, the schools in 
this study are all located in urban districts and primarily serve black and Hispanic populations. 
They also have high numbers of students who qualify for free or reduced-price lunch, suggest- 
ing that the majority of the students in these schools come from low-income families. Although 
the random assignment design should create groups of schools that are comparable, a check of 
the equivalency of school characteristics before implementation was done in order to ensure that 
random assignment resulted in a similar distribution of characteristics across the DN and non- 
DN schools. 2 

Table 2.1 compares DN and non-DN schools during the year before implementation on 
the ABC indicators (attendance, behavior, and course performance) that the study uses as inter- 
mediate outcome measures. Across the full sample, no statistically significant differences were 
observed on any measures between the DN and non-DN schools, indicating that before the 
rollout of Diplomas Now, the schools were equivalent on these outcomes, and that any differ- 
ences that are observed between the two groups of schools post-random assignment were 
caused by the implementation of the Diplomas Now model, and not by preexisting differences 
between schools. 3 

Table 2.2 shows baseline comparisons for students in the analytic sample used for the 
impact analyses. Since the primary impacts in this report were calculated for the second cohort 
of students to be enrolled in DN schools after implementation, these tables present information 
only on students in that cohort. Because Table 2.2 compares students rather than schools at 
baseline, figures are calculated from student records that provide data about students before 
their entry into a study school. For example, the test score measures are calculated from stu- 
dents’ performance on state standardized tests in the year before implementation, in fifth and 
eighth grades, effectively providing a measure of performance that is as close to the beginning 
of the school year as possible. 

Table 2.2 indicates that there were no statistically significant differences between stu- 
dents enrolled in DN and non-DN schools, with the exception of Hispanic students, of which 
there were significantly more in DN schools. In addition, a statistical test of all race/ethnicity 


2 For additional details about the characteristics of the study schools at the time of random assignment, 
see the first report from this evaluation (Corrin et al., 2014). 

3 For this report, a finding is described as statistically significant if there is a 10 percent probability or less 
that it was the result of chance. 
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Table 2.1 


Baseline Comparison of DN and Non-DN Schools 
on Attendance, Behavior, and Course Performance 



DN 

Schools 

Non-DN 

Schools 

Estimated 

Difference 

P- Value 

Attendance 

Percentage of enrolled days attended 

88.1 

88.2 

0.0 

0.959 

Percentage of students who attended over 

90 percent of enrolled days 

65.6 

67.5 

-1.9 

0.558 

Percentage of students who attended over 

85 percent of enrolled days 

77.3 

77.7 

-0.3 

0.897 

Behavior 

Percentage of enrolled days suspended or expelled 

2.0 

1.9 

0.2 

0.675 

Percentage of students who were ever 
suspended or expelled during year 

30.9 

27.2 

3.7 

0.213 

Percentage of students who were suspended or 
expelled for 3 or more days 

22.2 

19.6 

2.6 

0.386 

Course performance 3 

Percentage of core courses passed 

82.3 

82.0 

0.3 

0.800 

Percentage of students who had no core 
course failures during year 

66.9 

65.1 

1.8 

0.381 

Percentage of students who had no math course 
failures during year 

79.0 

76.2 

2.8 

0.101 

Percentage of students who had no English/ 
language arts course failures during year 

82.6 

80.7 

1.9 

0.335 

ABC composite measures' 3 

Percentage of students above stability threshold 

43.0 

42.5 

0.5 

0.873 

Percentage of students with no early warning 

indicators 

55.7 

54.4 

1.4 

0.700 

Sample size 

29 

29 




(continued) 
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Table 2.1 (continued) 


SOURCE: MDRC calculations based on sixth- and ninth-grade student records obtained from school districts 
for the year prior to the implementation of Diplomas Now (2010-201 1 for wave 1 schools and 201 1-2012 for 
wave 2 schools). 

NOTES: One DN school was dropped from the course performance analysis because baseline course data were 
not available for that school. 

Difference estimates are regression adjusted, controlling for blocking of random assignment. A two-tailed t- 
test is used for all statistical tests presented in this table. Statistical significance levels are indicated as follows: 
*** = i percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Core courses include all math. English/language arts, science, or social studies courses each student 
attempted during the school year. 

b The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 
district, was never suspended or expelled, and did not fail any core courses attempted during the school year. 
The “no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the 
district, was suspended or expelled for fewer than three days, and did not fail any math or English/language arts 
courses during the school year. If a student has missing course data but meets the other two criteria for an ABC 
composite measure, the student is assigned a value of 1 . 


variables combined, as well as a combined test of all variables in the table, did not find signifi- 
cant differences. 4 Thus, aside from one difference in ethnic composition between DN and non- 
DN schools, the two groups of schools and students were similar before the intervention. 


Data Sources 

All data used for impact analyses were obtained from individual administrative student records 
collected from each of the 1 1 participating school districts. In addition to demographic infor- 
mation about each student, including race, gender, and free or reduced-price lunch eligibility, 5 
these records included information on student days of enrollment, attendance, and school sus- 
pensions and expulsions. Furthermore, all student records included all courses students were 
enrolled in and their final grades. Finally, for years in which students were enrolled in tested 
grades, the data included both scaled scores and proficiency levels on state-administered Eng- 
lish language arts (ELA) and math exams. 


4 A chi-square test was used. A test of all variables is used because conducting many individual tests in- 
creases the risk of a false positive. 

Alter DN implementation began, one district began a program in some schools in which all students re- 
ceived free lunch, and parents no longer had to turn in paperwork establishing eligibility. Beginning with this 
cohort year, all students in those schools are marked as free-lunch eligible, so the measure is not usable as a 
proxy measure for income status. There are four schools in this district. 
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Table 2.2 


Baseline Comparison of DN and Non-DN Schools, 
Characteristics of Students in Cohort 2 Analysis Sample 


Characteristic (%) 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Difference 

P- Value 

Eligible for free/reduced-price lunch 

89.9 

89.1 

0.8 

0.525 

Race/ethnicity a 

Black 

54.0 

61.7 

-7.6 

0.195 

Hispanic 

36.7 

27.6 


0.029 

Asian 

3.7 

4.9 

-1.3 

0.399 

White 

4.2 

4.6 

-0.3 

0.863 

Other 

1.3 

1.2 

0.2 

0.811 

Gender 

Male 

54.7 

54.3 

0.4 

0.691 

English language learners 

18.0 

17.3 

0.7 

0.737 

Special education status 

19.2 

17.1 

2.1 

0.119 

Overage for assigned grade 

32.0 

31.9 

0.1 

0.973 

Proficient on state tests in prior year 

English/language arts 

36.3 

38.9 

-2.6 

0.269 

Math 

35.0 

38.0 

-3.0 

0.158 

Sample size 

29 

29 




SOURCE: MDRC calculations based on student records obtained from school districts of students attending sixth 
and ninth grades during the second year of implementation (2012-2013 for wave 1 schools and 2013-2014 for 
wave 2 schools). 


NOTES: Difference estimates are regression adjusted, controlling for blocking of random assignment. A two- 
tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are indicated as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

An omnibus test of all baseline characteristics included in this table did not find a statistically significant 
overall difference between DN and non-DN schools. 

a A chi-square test of all race/ethnicity categories combined did not find a statistically significant overall 
difference between DN and non-DN schools. 
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In addition to collecting administrative student records, the research team fielded ad- 
ministrator, teacher, and student surveys in all study schools in the springs of 2012, 2013, and 
2014. The surveys were designed to collect student and staff views on issues such as the school 
environment, the climate of teaching and learning, levels of bullying and other behavioral mat- 
ters, and community relations. 


Measures 

ABC Outcomes 

Central to the Diplomas Now theory of action is the development and regular monitor- 
ing in schools of a set of early warning indicators that alert school leaders and staff members to 
students who are off track or at risk of falling off track on ABC outcomes (attendance, behavior, 
and course performance). The ABC outcomes are considered to be intermediate outcomes that 
are critical to the long-term program goals of more students graduating from high school who 
are college and career ready. Therefore the focus is on these three outcomes, as well as an over- 
all composite measure of students being on track to graduate, as indicative of whether Diplomas 
Now is having a positive impact on students’ likelihood of achieving the longer-tenn goals of 
the model. 

The measures of attendance, behavior, and course performance used in the impact 
analyses for this report are described in more detail in Table 2. 3. 6 These outcomes all capture 
the average change across all students. The evaluation’s primary research questions are about 
the impact of Diplomas Now implementation relative to schools’ typical school improvement 
efforts: 

• What is the impact of Diplomas Now on students’ attendance rates (propor- 
tion of enrolled days attended) during the sixth- or ninth-grade school year? 

• What is the impact of Diplomas Now on the proportion of enrolled days sus- 
pended (in or out of school) or expelled during students’ sixth- or ninth-grade 
year? 

• What is the impact of Diplomas Now on the proportion of attempted core 
courses passed by the end of students’ sixth- or ninth-grade year? 


6 The primary measures were identified before the start of the study as likely to be affected by the Diplo- 
mas Now theory of action. Preidentification of these measures is one of the strategies used to prioritize out- 
comes for determining the effectiveness of the Diplomas Now model and help protect against spurious find- 
ings that might arise from conducting multiple hypothesis tests. 
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Table 2.3 

Description of ABC Outcome Measures 


Outcome 

Attendance 

Percentage of enrolled days attended' 1 


Percentage of students who attended 
over 90 percent of enrolled days' 5 

Percentage of students who attended 
over 85 percent of enrolled days 

Behavior 

Percentage of enrolled days 
suspended or expelled' 1 

Percentage of students who were ever 
suspended or expelled during year 

Percentage of students who were 
suspended or expelled for 3 or 
more days 

Course performance 

Percentage of core courses passed 1 


Percentage of students who had no 
core course failures during year 
Percentage of students who had no 
math course failures during year 


Description 


Proportion of the days that a student was enrolled in 
the school district on which that student was present 
at school 

For each student, the value is 1 if the student attended 
over 90 percent of enrolled days and 0 otherwise. 

For each student, the value is 1 if the student attended 
over 85 percent of enrolled days and 0 otherwise. 


Proportion of the days that a student was enrolled in 
the school district for which that student was 
suspended or expelled 

For each student, the value is 1 if the student was ever 
suspended or expelled during the school year and 0 
otherwise. 

For each student, the value is 1 if the student was 
suspended or expelled for 3 or more days and 
0 otherwise. 


Proportion of the core English/language arts, math, 
science, and social studies classes a student took 
during the year that the student passed 
For each student, the value is 1 if the student did not 
fail any core courses during the year and 0 otherwise. 
For each student, the value is 1 if the student did not 
fail any math courses during the year and 0 otherwise. 


Primary 

Outcome? 


Yes 

No 

No 


Yes 


No 


No 


Yes 

No 

No 


Percentage of students who had no 
English/language arts course 
failures during year 

ABC composite measures 

Percentage of students above stability 
threshold 

Percentage of students with no early 
warning indicators 


For each student, the value is 1 if the student did 
not fail any English/language arts courses 
during the year and 0 otherwise. 


For each student, the value is 1 if the student attended over 
90 percent of enrolled days, was never suspended or 
expelled, and passed all core courses. 

For each student, the value is 1 if the student attended over 
85 percent of enrolled days, had fewer than 3 days of 
suspensions or expulsions, and passed all math and 
English/language arts courses. 


No 


No 


No 


NOTES: “Enrolled days are used as the denominator rather than total days in the school year because the analysis 
sample includes students who did not attend school in the district for the entire school year. There is no statistically 
significant difference in enrolled days between DN and non-DN schools. 

b Students who attend school for less than 90 percent of days are typically labeled “chronic absentees.” 
c Focus was placed on passing all core courses because these are the courses that students generally need to 
complete to be promoted to the next grade and to graduate. 
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In addition to these primary outcomes, which are measured on a continuous scale, sev- 
eral threshold indicators of student success are used in exploratory impact analyses. These 
measures capture two levels of the ABC outcomes, a more stringent “stability” threshold and a 
threshold tied to whether students have any early warning indicators. The stability threshold 
measures are based on whether students attended school more than 90 percent of the time, were 
not suspended or expelled during the study year, or passed all of their core classes. 7 These 
thresholds also represent nonnative expectations for students not to be chronically absent (typi- 
cally defined as missing 10 percent or more days of school), not to get in serious trouble, and 
not to fail classes. A composite of all three of these threshold measures of success indicates 
whether a student is defined in this report as being on a stable trajectory to graduation. 

The second level of thresholds on the ABC outcomes measures whether Diplomas Now 
is making incremental progress with students by reducing the number with any early warning 
indicators, which are suggestive of being off the track to graduation. These thresholds align with 
levels used by Diplomas Now and school staff members in their day-to-day practice for as- 
sessing student need for additional intervention. These early warning indicators are defined as 
85 percent or lower attendance, more than one suspension or disciplinary action, 8 or Eng- 
lish/language arts or math course failure — the two subject areas for which Diplomas Now pro- 
vides additional support to teachers and students. Analyses of the percentages of students above 
and below these early warning thresholds allow for the assessment of the impacts of Diplomas 
Now on the number of students who are in early warning status in terms of each of the ABC 
outcomes as well as overall across all three. All of these outcomes are described in Table 2.3. 

Early Outcomes (Survey Measures) 

As illustrated in Figure 1.1, the early school and student outcomes described in this sec- 
tion are predicted to be affected by Diplomas Now program activities. Effects on these out- 
comes are hypothesized to lead to effects on the ABC outcomes. Chapter 3 presents analyses on 
data collected from surveys of administrators, teachers, and students about these outcomes. 

Early School Outcomes 

Through their survey responses, teachers reported on their perception of the school cli- 
mate, including whether the school environment was conducive to teaching and learning and 


7 Diplomas Now provides specific support for teachers and students in English/language arts and math, 
but graduation from high school is dependent on earning credits across core course areas that typically in- 
clude science and social studies (or history) as well. 

8 Because data on the number of disciplinary actions were not provided across all districts, analyses con- 
ducted for this report use three or more total days suspended or expelled as a proxy for more than one disci- 
plinary action. 


21 



whether there were consistently enforced rules for student behavior. Students reported whether 
they felt safe and happy at school and whether the school had problems with serious student 
behavior issues (such as bullying, fighting, physical or verbal abuse of teachers by students, 
drug and alcohol abuse by students, and gang activity). Teachers and administrators both re- 
ported on parent and community involvement, including whether parents and community mem- 
bers volunteered at the school or were involved in school activities and whether there was effec- 
tive communication between the school and parents and community members. Students also 
reported on their participation in after-school activities, especially those that included academic 
support. 9 


Early Student Outcomes 

Students were also asked specific questions about their attitudes and behaviors as 
measures of their confidence, engagement and commitment to school, effort and persistence in 
school, study habits, and relationships with adults and peers. 10 


Analytic Sample 

Sample Used for Impact Analysis 

The main analytic sample includes first-time sixth- and ninth-grade students attending 
the study schools during the second year of implementation after random assignment (Cohort 
2). 1 1 The key assumption behind choosing this group as the primary analysis sample is that the 
Diplomas Now model is a multiyear and multifaceted program. Therefore, it may take schools 
more than a year to adjust to the program and figure out how implementation works best within 
their individual sites. First-time sixth- and ninth-graders are chosen because the model includes 
intensive supports for students during their first year attending middle or high school, and some 
of those supports lessen in intensity as students enter higher grade levels. 12 Thus measuring im- 
pacts at the end of the year in which the second cohort of students made the transition into the 


9 The surveys did not measure the effects of Diplomas Now on the outcome of increased stakeholder 
communication. Given the importance of the Diplomas Now staff members as stakeholders at the DN 
schools, it was hard to find comparable measures of communication between the DN and non-DN schools. 

10 See Appendix A for a description of the construction of all early school and student outcomes that are 
made up of more than one survey item. 

n For one random assignment block with three schools, the data in the main analysis is from sixth-grade 
students who attended the schools in the first year of implementation, because two of the schools did not have 
sixth-grade students in the second implementation year. 

I2 A11 high schools are ninth through twelfth grade. There are a few middle schools that are kindergarten 
through eighth or fifth through eighth grade. In these cases, Diplomas Now still focused efforts on sixth- 
graders. 


22 



school may present a truer picture of the impact of Diplomas Now on student outcomes than 
would analyses at the end of the first year of implementation. That said, examining only the 
second year of implementation may not be enough to show the full effects of a program that is 
designed to create change for students in multiple grades across multiple years. 

While many of the schools in the analysis sample have large portions of repeating stu- 
dents, particularly students repeating ninth grade, the decision was made to focus on first-time 
students because repeaters really belong to the prior cohort of entering students. Regarding the 
primary analysis sample, students who are repeaters in the second implementation year are 
members of the first cohort of study students, and thus already have their sixth-grade or ninth- 
grade outcomes analyzed with those of their cohort peers; their full trajectory will be included in 
the longer-term analyses to be discussed in subsequent reports. The analysis of first-time-in- 
grade students therefore addresses how well the Diplomas Now model keeps students on track 
as they move through secondary school. Longitudinal analysis that follows students to gradua- 
tion or successful high school transition will better capture the impacts of the full intent of Di- 
plomas Now to influence students’ longer-term school success. 

Findings are presented for all students in Cohort 2, as well as separately for students in 
middle and high schools. As mentioned, also presented in this report are outcomes for the first 
cohort of sixth- and ninth-grade students to enroll in study schools after random assignment. 13 
In addition, subgroup analyses are included for high school students who were or were not on a 
stable trajectory to graduation in eighth grade; middle school students who scored above and 
below proficient on state standardized tests at the end of elementary school; and students who 
remained enrolled in study schools for the entire school year and therefore fully experienced the 
programming offered at their school, whether it was the Diplomas Now model in DN schools or 
other practices or reforms being implemented in the non-DN schools. 

It is important to note that the main analytic sample (that is, first-time sixth- and ninth- 
graders in Cohort 2) represents the policy-relevant sample because it includes students who 
were enrolled in the school at any time during the year, regardless of how long they attended. 
This sample presents a truer picture of the kinds of students who enroll in study schools that 
serve mobile populations and represents impacts for a program designed to serve all students in 
a school, regardless of whether they remain for the entire year. 


13 Cohort 1 includes three schools in one random assignment block that served only seventh and eighth 
grades. For these three schools, seventh-grade student outcomes are included in the analyses because seventh 
grade was the transition year into middle school. Of those three schools, two began offering sixth grade in the 
second year of implementation, and one closed and is not included in the Cohort 2 analyses. 
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Sample Used for Analysis of Early Outcomes 

The samples of teachers and administrators included in the analyses of the early school 
outcomes were chosen to match the analyses of the primary student ABC outcomes. With that 
goal in mind, the staff surveys were fielded in the spring of the second year after Diplomas Now 
implementation began, and only teachers who taught sixth- and ninth-grade students are includ- 
ed in the analyses. The findings provide context for understanding how the faculty and admin- 
istration viewed initial changes in their schools that occurred as a result of the implementation 
of the Diplomas Now model. 

The early school and student outcomes measured using the student survey come from 
the survey of sixth- and ninth-grade students fielded during the spring of the first year after im- 
plementation, so these outcome measures reflect the opinions and feelings of the students in 
Cohort l. 14 


Analysis 

A two-level statistical model was used to estimate the impacts of the Diplomas Now model, 
generating estimates at both the block level (since schools were randomized within blocks) and 
overall for the full sample of schools. Block-level estimates essentially provide the level of dif- 
ference for a given outcome between DN and non-DN schools within a block. The estimated 
impact for the overall sample is an average of the effects for each block, weighted by the num- 
ber of DN schools within the block. The impact thus represents the average outcome for the 
average student enrolled in the average DN school. However, it is important to note that the dif- 
ference presented in the impact tables represents the estimated impact rather than the true im- 
pact, because although schools were randomly assigned to implement Diplomas Now, there is 
still a possibility that differences could be observed by chance. Further technical details of the 
analysis are available in Appendix A, including school- and student-level characteristics con- 
trolled for in the analysis and the hierarchical linear regression model. Results from the impact 
analysis on early outcomes based on survey data are in Chapter 3. Results from the impact anal- 
ysis on intermediate ABC outcomes are presented in Chapter 4. Box 2.1 explains how to inter- 
pret the findings reported in the tables in both of these chapters. 


14 The second cohort of students, sixth- and ninth-grade students attending the study schools during the 
second year of implementation, was not surveyed. 
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Box 2.1 


Understanding the Impact Tables in This Report 

The tables in Chapters 3 and 4 display the impacts of the Diplomas Now model on school and 
student outcomes. These tables also present the average outcomes for students, teachers, and 
administrators in the study schools, to provide context for inteipreting the magnitude of the im- 
pact findings. The values presented in these tables are derived as follows: 

Estimated difference or estimated impact. This column shows the estimated difference be- 
tween DN and non-DN schools with respect to the attendance, behavior, and course perfor- 
mance outcomes of their students or the survey responses of their students, teachers, and admin- 
istrators. The values in this column are obtained by estimating the difference in outcomes for 
each random assignment block, and then adjusting, or weighting, each of these estimates by the 
number of DN schools in the block, so that each school contributes equally. Because of this 
weighting procedure, the average estimate presented in the table represents the difference (or 
impact) for the average school in the DN group. The statistical significance of the estimated dif- 
ference is indicated when the p-value is less than or equal to 10 percent (*** = 1 percent; ** = 5 
percent; * = 10 percent). A p-value of 10 percent means that if there truly were no difference be- 
tween DN schools and non-DN schools on a particular outcome, the probability of finding the 
estimated difference would be quite small — no more than 10 percent. Therefore, probability 
suggests that there is an actual difference, and the number in the impact column is an estimate of 
what that difference is. 

DN schools. This column shows the observed mean outcome for schools in the DN group. As 
with the differences above, these mean outcome levels are obtained by weighting the mean out- 
come level in each random assignment block by the number of DN schools in that block. Thus 
the mean outcome represents the mean outcome for the average school in the DN group. 

Non-DN schools. This column provides an estimate of what the mean outcome for the average 
DN school would have been had it not been randomly assigned to implement the DN model. 
The values in this column are the mean outcomes for schools in the non-DN group, adjusted to 
reflect the observed distribution of DN schools across random assignment blocks. 

Effect size. This column shows the estimated impact translated into an effect size. The effect 
size is the estimated impact divided by the standard deviation of the outcome for the non-DN 
group. For example, an effect size of 0.20 represents 20 percent of the standard deviation for 
that outcome. This measure allows comparison between outcome measures that are on different 
scales. 


25 






Chapter 3 

Early School and Student Outcomes 


As illustrated in the logic model in Chapter 1 , Diplomas Now is hypothesized to achieve its in- 
termediate goals of improving attendance, behavior, and course performance (ABC outcomes) 
through several mediating pathways. These pathways, or early outcomes, include positive 
school climate, the addition of academic after-school activities, and increased parent and com- 
munity support, along with student attitudes and behaviors, including self-confidence, engage- 
ment and effort in school, study habits, and relationships with adults and peers. 1 As noted in 
Chapter 2, the analyses of the early school and student outcomes are based on data collected 
from administrator, teacher, and student surveys. Survey responses at schools randomly as- 
signed to implement Diplomas Now (DN schools) were compared with the responses from the 
same respondent groups in the schools assigned not to implement the program (non-DN 
schools). The responses from administrators and teachers come from surveys administered at 
the end of the second year of implementation. Administrators include school principals and as- 
sistant principals. Teacher respondents include sixth- and ninth-grade teachers at the schools. 2 
Since sixth- and ninth-grade students are the focus of the analyses in this report and students in 
sixth and ninth grades were surveyed only in the spring of the first year of implementation, the 
early school and student outcomes reported using student survey responses come from the first 
year of implementation. 3 

The following are key points from the early school and student outcome findings: 

• The Diplomas Now model had a positive and statistically significant impact 
on teachers’ perceptions of school climate. Although there were no other sta- 
tistically significant impacts on early school outcomes as reported by admin- 
istrators and teachers, the findings across all the climate- and community- 
oriented outcomes tend to be positive. 

• Students at DN schools reported participating in more academically focused 
after-school activities than students at non-DN schools. Student perceptions 
of other outcomes, including school safety and climate and behavioral issues 


'One of the early school outcomes listed in Figure 1.1, increased stakeholder communication, is not repre- 
sented in these analyses because the Diplomas Now school-level staff members represent a set of stakeholders, 
and comparable stakeholders do not necessarily exist in non-DN schools. 

2 Seventh-grade teachers’ responses are included for schools that did not have sixth grade in the second 
year of implementation. 

3 Seventh-grade students’ responses are included for schools that did not have a sixth grade in the first year 
of implementation. 
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at school, were similar for students at DN and non-DN schools during the 
first year of implementation. 

• The Diplomas Now model did not have an effect on students’ self- 
perceptions and school behaviors as measured by the student survey during 
the first year of implementation. 

• Students at DN schools were more likely to report a positive relationship 
with an adult at school who was not a teacher, but there was no difference in 
student perceptions of relationships with teachers, administrators, and other 
students at DN schools compared with non-DN schools. 


Findings on Early School Outcomes 

Table 3.1 focuses on teachers and administrators’ perceptions of school climate and parent and 
community involvement at the school. (See Box 2.1 for an explanation of how to read the im- 
pact tables.) The Diplomas Now model had a positive and statistically significant impact on 
how strongly teachers agreed that the climate at their school was conducive to teaching and 
learning, suggesting that by the second year of implementation, the Diplomas Now model was 
positively affecting teachers’ perceptions of their school environment. 

Although there was only one statistically significant finding on early school outcomes 
reported by administrators and teachers, there is a general positive pattern in the findings; all but 
one of the estimated differences in Table 3.1 is positive. This is encouraging especially for the 
outcomes related to increased parent and community involvement, considering that there was 
little difference in service during the second year between DN and non-DN schools on efforts 
by schools to plan for and offer more opportunities for parents and community members to par- 
ticipate. 4 The one negative but not statistically significant outcome in this area is on the average 
number of times per month parents volunteered in the classroom, according to administrator 
reports. Given the influx of student support from AmeriCorps members in the City Year pro- 
gram, it may not be surprising if DN schools are less likely to incorporate parent volunteers as 
well — especially during the early years of implementation, when schools are still figuring out 
how to employ the AmeriCorps volunteers in classrooms most efficiently and effectively. 

Table 3.2 displays the impact of the Diplomas Now model on sixth- and ninth-grade 
students’ perceptions of school climate and their participation in after-school activities at the 
end of the first year of implementation. There are no statistically significant impacts on stu- 
dents’ feelings of safety at school, their enjoyment of school, or their reports of serious student 


4 See Sepanik et al. (2015), pp. 38-41. 
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Table 3.1 


Early School Outcomes, Teacher and Administrator Responses, 
Implementation Year 2 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Difference 

Effect 

Size 

P-Value 

More positive school climate 






Teachers reported the school environment was 
conducive to teaching and learning. 

(0 = strongly disagree, 10 = strongly agree) 

6.47 

6.07 

0.40 

0.14 * 

0.096 

Teachers reported rules for student behavior were 
consistently reinforced by administrators and 
other teachers. (0 = never, 5 = sometimes, 

10 = always, as needed) 

6.75 

6.65 

0.10 

0.04 

0.647 

Increased parent and community involvement 






Teachers reported their school had effective 
communication with and participation from students' 
families and the local community. (0 = not offered, 5 = fair, 
10 = excellent) 2 

6.39 

6.20 

0.19 

0.08 

0.370 

Teachers reported parents were involved in school 
activities and supported school reform efforts. 

(0 = strongly disagree, 10 = strongly agree) 

4.87 

4.59 

0.28 

0.11 

0.129 

Teachers reported community members were involved in 
school activities and supported school reform efforts. 

(0 = strongly disagree, 10 = strongly agree) 

5.42 

5.17 

0.26 

0.11 

0.213 

Administrators reported parents were involved in school 
activities and supported school reform efforts. 

(0 = strongly disagree, 10 = strongly agree) 

6.51 

5.88 

0.63 

0.25 

0.137 

Administrators reported community members were 
involved in school activities and supported school reform 
efforts. (0 = strongly disagree, 10 = strongly agree) 

6.70 

6.53 

0.17 

0.07 

0.663 

Teachers reported parents worked as volunteers in 
classes, (average times per month)” 

1.25 

0.83 

0.42 

0.13 

0.274 

Teachers reported community members worked as 
volunteers in classes, (average times per month)” 

1.24 

1.01 

0.23 

0.08 

0.548 

Administrators reported parents worked as volunteers in 
classes, (average times per month)” 

3.89 

4.47 

-0.58 

-0.09 

0.701 

Administrators reported community members worked as 
volunteers in classes, (average times per month)” 

4.39 

4.00 

0.39 

0.07 

0.772 

Sample size 

31 

30 





(continued) 
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Table 3.1 (continued) 


SOURCE: Follow-up surveys of teachers and administrators (principals and assistant principals) administered 
during the school years of 2012-2013 and 2013-2014. Respondents included middle school teachers who taught 
sixth grade (or seventh grade for schools not serving sixth grade) and high school teachers who taught ninth grade. 

NOTES: Across 60 study schools, 1,339 teachers participated in the follow-up survey. One study school stopped 
serving sixth grade after the first year of implementation and is not included in this analysis. For each of the 
outcome measures, data are missing for no more than 5.5 percent of the teachers. The difference in the percentage 
of missing data between DN and non-DN schools is no more than 2.4 percentage points for any of the outcome 
measures. 

Across 61 study schools, 170 administrators participated in the follow-up survey. For each of the above 
measures, data are missing for no more than 7.1 percent of the administrators. The difference in the percentage of 
missing data between DN and non-DN schools is no more than 2.2 percentage points for any of the above 
measures. 

Effect sizes were computed using the standard deviations of all non-DN school respondents for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

“Markow and Pieters (2012). 

b This outcome is calculated by weighting the frequency per month and then summing the weighted survey items 
in order to get a measure of total times per month. For example, 0 = never, 1 = at least once a month, 2 = more than 
once a month but not weekly, 5 = once a week, 12 = more than once a week but not daily, and 20 = daily. 

behavior issues such as bullying, fighting, physical and verbal abuse, drug and alcohol abuse, or 
gang activity at school. This suggests that the Diplomas Now model had little effect on student 
perceptions of school climate during the first year of implementation. It is possible that it could 
take more than one year for the Diplomas Now model to take hold at a school in a way that 
changes the overall climate of the school and the level of behavioral problems, as well as stu- 
dents’ feelings of safety and ability to enjoy school. As noted above, second-year surveys re- 
vealed an impact on teachers’ perceptions of school climate. 

Students at DN schools did report participating in more after-school activities, including 
those that offered homework help, test preparation, and tutoring, than students at non-DN 
schools. One of the services under the third pillar of the Diplomas Now model, Tiered Student 
Supports, is an after-school program run by the City Year AmeriCorps members, open to all 
students but targeted to students struggling academically. (See Figure 1.1.) In this way, the Di- 
plomas Now model is set up to directly affect this outcome, and it succeeded in this goal during 
the first year. 


Findings on Early Student Outcomes 

Table 3.3 displays the early student outcomes during the first year of implementation. The di- 
rection of the effects is inconsistent and there is only one statistically significant impact, 
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Table 3.2 


Early School Outcomes, Sixth- and Ninth-Grade Student Responses, 

Implementation Year 1 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Difference 

Effect 

Size 

P-Value 

More positive school climate 






Students felt safe at school. (0 = strongly disagree, 

3 = strongly agree) 

1.73 

1.75 

-0.01 

-0.02 

0.795 

Students enjoyed coming to school. 

(0 = strongly disagree, 3 = strongly agree) 

1.77 

1.80 

-0.03 

-0.04 

0.355 

Students would switch to a different school if they 
were given the choice. (0 = strongly disagree, 

3 = strongly agree) 

1.55 

1.53 

0.02 

0.02 

0.652 

The school had problems with bullying/cyber 
bullying, students fighting, students cutting class, 
and teachers not being able to control the 
classroom. 

(0 = not a problem, 3 = big problem) 

1.81 

1.85 

-0.04 

-0.05 

0.424 

The school had problems with students physically 
or verbally abusing teachers or other adults. 

(0 = not a problem, 3 = big problem) 

1.15 

1.12 

0.03 

0.03 

0.560 

The school had problems with destruction of 
school property, students bringing weapons to 
school, students abusing drugs or alcohol, 
and gang activity. 

(0 = not a problem, 3 = big problem) 

1.36 

1.40 

-0.04 

-0.04 

0.487 

After-school activities 






Students participated in school-sponsored 
after-school activities. (%) 

56.7 

51.2 

5.4 

o n ** 

0.027 

Students participated in after-school activities such 
as homework help/assignment completion, test 
preparation, or tutoring. (%) 

25.1 

17.3 

7.8 

0.20 *** 

<0.001 

Sample size 

32 

30 





(continued) 


suggesting that the Diplomas Now model had little effect during the first year on sixth- and 
ninth-grade students’ attitudes and behaviors related to school. It may be that model compo- 
nents needed to be implemented at a higher level than they were during the first year in order to 
elicit effects on these outcomes. Another possibility is that students’ attitudes and behaviors in 


31 







Table 3.2 (continued) 


SOURCE: Follow-up surveys of students administered during the school years of 201 1-2012 and 2012- 
2013. 

NOTES: Across 62 study schools, 9,356 students participated in the follow-up survey. For each outcome 
measure, data are missing for no more than 5.3 percent of the students, except for the “students reported 
participating in school-sponsored after-school activities’’ item and the “students reported participating in 
after-school activities such as homework help/assignment completion, test preparation, and tutoring” item, 
where data for 1 1.5 percent of students are missing. The difference in the percentage of missing data 
between DN and non-DN schools is no more than 1 percentage point for any of the outcome measures. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 
are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 


school may be particularly tough to change and may require persistent effort over a longer peri- 
od of time. 

Although students at DN schools did not report stronger relationships with teachers, 
administrators, or other students than did students at non-DN schools, they were more likely 
than their counterparts at non-DN schools to report that they had a positive relationship with an 
adult at the school who was not a teacher. This finding aligns with the implementation of the 
Diplomas Now model, which creates more opportunities for students to interact with and be 
supported by adults who are not teachers. Specifically, City Year brings a group of 8 to 15 
AmeriCorps members into the schools to work directly with students identified as struggling on 
one or more of the ABC outcomes. In addition, Communities In Schools case managers assess 
the needs of students who are struggling the most and connect them with specialized interven- 
tions, such as group or individual counseling and tutoring, which may be provided by adults 
other than teachers. 


Conclusion 

Although the Diplomas Now model has shown few statistically significant impacts on early 
school outcomes, the results suggest that the model is having an effect on teachers’ perceptions 
of school climate by the second year of implementation. It is important to note that changes in 
teachers’ perceptions are likely to be an important step toward changing overall school climate 
and students’ perceptions, which may take longer. The model also had an impact on some early 
school and student outcomes, including students’ participation in after-school activities and rela- 
tionships with adults at the school who were not teachers, that can be traced to Diplomas Now 
model components. Many of the student outcomes for which Diplomas Now did not produce 
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Table 3.3 


Early Student Outcomes, Sixth- and Ninth-Grade Student Responses, 

Implementation Year 1 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Difference 

Effect 

Size 

P- Value 

Greater confidence and self-worth 






Students believed if they tried hard, did not give up, 
and had enough time, they could do their schoolwork 
well. (0 = strongly disagree, 3 = strongly agree) 

2.39 

2.38 

0.01 

0.02 

0.541 

Students worried about projects, tests, poor grades, 
and schoolwork in general. 

(0 = strongly disagree, 3 = strongly agree) 

1.77 

1.77 

0.00 

0.00 

0.962 

Increased encasement and commitment 






Students thought their work was interesting and 
liked what they learned in class. 

(0 = never, 3 = all the time) 

1.53 

1.57 

-0.04 

-0.05 

0.323 

Students tried less at school over time, not caring 
about school, and not participating in class. 

(0 = strongly disagree, 3 = strongly agree) 

0.81 

0.82 

-0.01 

-0.02 

0.612 

Students believed learning and working hard in 
class was important for their future. 

(0 = strongly disagree, 3 = strongly agree) 

2.25 

2.26 

-0.01 

-0.02 

0.678 

Increased effort and persistence 






Students did not give up easily when they did not 
understand or when homework was difficult. 

(0 = strongly disagree, 3 = strongly agree) 

2.04 

2.03 

0.01 

0.01 

0.758 

Students paid attention, stayed on task in class, 
and completed all their schoolwork. 

(0 = never, 3 = all the time) 

1.84 

1.84 

0.00 

0.00 

0.982 

Improved study habits and strategies 






Students used study strategies such as note taking, 
graphic organizers, and formulas to help remember 
information. (0 = never, 3 = all the time) 

1.67 

1.70 

-0.02 

-0.04 

0.467 

Students used social skills and conflict resolution 
strategies for controlling anger. 

(0 = never, 3 = all the time) 

1.49 

1.49 

0.00 

0.00 

0.969 


(continued) 
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Table 3.3 (continued) 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Difference 

Effect 

Size 

P- Value 

Positive relationships with adults and Deers 






Students had positive relationships with teachers. 3 
(0 = strongly disagree, 3 = strongly agree) 

1.65 

1.69 

-0.04 

-0.06 

0.316 

Administrators were respectful of students and 
teachers and were fair when enforcing rules. 

(0 = strongly disagree, 3 = strongly agree) 

1.90 

1.86 

0.04 

0.05 

0.373 

Students had a positive relationship with at least 
one adult in the school other than a teacher. b 
(0 = strongly disagree, 3 = strongly agree) 

1.85 

1.77 

0.08 

o n ** 

0.011 

Students felt they fit in at school and that other 
students at school accepted them for who they 
were. (0 = strongly disagree, 3 = strongly agree) 

1.91 

1.91 

0.00 

-0.01 

0.866 

Sample size 

32 

30 





SOURCE: Follow-up surveys of students administered during the school years of 2011-2012 and 2012- 
2013. 

NOTES: Across 62 study schools, 9,356 students participated in the follow-up survey. For each outcome 
measure, data are missing for no more than 5.7 percent of the students. The difference in the percentage of 
missing data between DN and non-DN schools is no more than 1 percentage point for any of the outcome 
measures. 

Effect sizes were computed using the standard deviations of all non-DN school students for the 
respective measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 
are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

‘This composite measure includes the following items: teachers got along with students, met with 
students to talk about schoolwork and problems outside of the classroom, really listened to students, set a 
positive example, praised students for good work, and tried to be fair. 

b This composite measure includes the following items: at least one adult other than a teacher really cared 
about students, checked in with students, encouraged students to do their best, and could help solve student 
problems at school or at home. 


statistically significant impacts during the first year involve changing attitudes, behaviors, and 
relationships. Although these outcomes are described as “early” in the logic model, it may take 
more than a year for these attitudes and relationships to fonn and build. These early school and 
student outcomes were theorized to lead to impacts on the ABC outcomes and, ultimately, 
graduation rates. The findings from analyses of the impacts of Diplomas Now implementation 
on the intermediate outcomes are discussed in the next chapter. 
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Chapter 4 

The Impact of Diplomas Now on Intermediate 
Outcomes of Student Attendance, Behavior, 
and Course Performance 


As described in Chapter 1 , the Diplomas Now partnership works with challenged schools in 
an effort to improve student attendance, to address and reduce behaviors that lead to discipli- 
nary action, and to improve student course performance, particularly successful completion of 
English/language arts and math courses. Attendance, behavior, and course performance are 
the “ABCs” that help predict whether students graduate or drop out. Prior reports from this 
national evaluation of Diplomas Now focused on the implementation of the whole-school re- 
form model over the first two years, and the previous chapter focused on how that implemen- 
tation affected early school and student outcomes using staff and student survey data. This 
chapter extends the story by looking at impacts on students’ ABC outcomes during their first 
year of middle or high school. 

The chapter first discusses the impacts on sixth- and ninth-grade students’ ABC out- 
comes for the full sample of schools in the study as well as looking at the high school and mid- 
dle school samples separately. The chapter then turns to related findings, including the impact 
of the Diplomas Now model on the percentages of students who reach specific thresholds on 
each individual ABC outcome, as well as a composite measure; the impacts on subgroups of 
students based on their preparedness before entering middle or high school; the differences in 
the impacts on sixth- and ninth-grade student outcomes after the first and second years of im- 
plementation; and the variation in impacts across the randomly assigned blocks of schools. To- 
gether, these findings offer a comprehensive view of the effects of the Diplomas Now model 
during the first two years of implementation. 

Ultimately, the goal of Diplomas Now is to get more students to graduate from high 
school. (See the long-term outcomes in Figure 1.1.) A future report from this evaluation will 
discuss whether implementation of the Diplomas Now model had an impact on ninth-graders’ 
high school graduation, as well as sixth-graders’ successful completion of their first year of high 
school. So while the focus of this report is on interim findings about the effect of Diplomas 
Now on students’ trajectory toward graduation, more follow-up is required to fully assess the 
model’s impact. 

Key points from this chapter include the following: 

• Implementation of Diplomas Now resulted in a positive, statistically signifi- 
cant impact on the percentage of sixth- and ninth-graders without any early 
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warning indicators — that is, students who had maintained an 85 percent at- 
tendance rate or better, were suspended fewer than three days, and passed 
English/language arts and math, the subject areas explicitly targeted for sup- 
port as part of the Diplomas Now model. There was also a small, nonsignifi- 
cant impact estimated on the percentage of students achieving a more strin- 
gent threshold of a 90 percent attendance rate, no suspensions or expulsions, 
and passing grades in four core content classes: EnglisManguage arts, math, 
history or social studies, and science. 

• The impacts on both these ABC composite measures increased from the first 
year of implementation to the second year of implementation, and by a statis- 
tically significant amount for the higher threshold indicator. This increase 
from Year 1 to Year 2 aligns with the increased service differentiation that 
occurred across the same time period (discussed in the review of implemen- 
tation findings in Chapter 1). 

• Continuous measures of the individual ABC outcomes — overall average at- 
tendance, behavior as measured by average number of days disciplined, and 
percentage of core courses passed — provide a snapshot of whether there are 
impacts on average across all students regardless of their position relative to 
the upper and lower thresholds. There were no statistically significant im- 
pacts on these measures. 

• On the threshold measures of the individual ABC outcomes, there were no 
statistically significant impacts on percentages of students achieving the at- 
tendance or course performance thresholds, although the impact estimates 
were positive in direction. Neither were there significant impacts on the per- 
centages of students disciplined at either threshold level, although estimates 
indicate that slightly higher percentages of students were disciplined in the 
DN schools. 

• Diplomas Now may have had a stronger impact in middle schools than in 
high schools. In middle schools, the impact of Diplomas Now implementa- 
tion resulted in statistically significant, positive impacts on the percentage of 
students with better than 90 percent attendance, the threshold below which 
students are considered “chronically absent,” and on the percentage of stu- 
dents achieving the lower composite threshold. There were small, positive, 
nonsignificant impacts estimated for the other attendance and composite 
measures and the course performance measures. The effects were consistent- 
ly more promising than those at the high school level. On average, Diplomas 
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Now did not produce statistically significant impacts at the high school level 
since outcomes improved similarly at DN and non-DN schools. 

• At the high school level, Diplomas Now was more successful at keeping 
students who had better than 90 percent attendance, no suspensions or ex- 
pulsions, and no core course failures in eighth grade above those thresholds 
in ninth grade than getting other students across them. 


Impacts on Continuous Measures of the ABC Outcomes 

Table 4.1 presents the impacts of the Diplomas Now model on continuous measures of attend- 
ance, behavior, and course performance for the second cohort of sixth- and ninth-grade students 
in the full analysis sample and separately for the middle school and high school samples. 1 Over- 
all, as shown in Panel A, Diplomas Now did not have a statistically significant positive or nega- 
tive impact on any of the three measures presented, and the impact estimates are small in size. 

Although Diplomas Now is designed to help both middle school and high school stu- 
dents stay, get back, or get on track, middle schools and high schools typically vary in size and 
structure and serve students at different places in their developmental and educational paths. In 
fact, some aspects of the Diplomas Now model differ between middle schools and high schools. 
Thus, impacts on student outcomes were also analyzed for these two groups of schools separate- 
ly, as presented in Panel B and Panel C of Table 4.1. There were no statistically significant im- 
pacts on these three ABC outcome measures at either level. However, the impact estimates at 
the middle school level appear more promising than those at the high school level. 


Impacts on Stability and Early Warning Indicator Thresholds for 
Attendance, Behavior, and Course Performance 

While continuous measures of the ABC outcomes provide information about whether Diplomas 
Now had an impact on average across all sixth- and ninth-grade students, Diplomas Now pro- 
gramming uses early warning indicators defined by attendance, behavior, and course perfor- 
mance thresholds to inform interventions for students and adjustments to practices within 
schools. As discussed in Chapter 2, students are defined as being on a stable trajectory or having 
no early warning indicators based on these thresholds. This section discusses the results of anal- 
yses of the percentages of students meeting stability thresholds on these measures (students on a 


'Please see Chapter 2 for a discussion of the analysis samples of schools and students, as well as descrip- 
tions of the outcome measures. Box 2.1 explains how to read the impact tables. 
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Table 4.1 


Impacts on Continuous Measures of 
Attendance, Behavior, and Course Performance, Cohort 2 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P-Value for 

Estimated 

Impact 

Panel A: Full sample 

Attendance 

Percentage of enrolled days attended 

89.7 

89.3 

0.4 

0.03 

0.602 

Behavior 

Percentage of enrolled days suspended or expelled 

1.5 

1.2 

0.3 

0.05 

0.177 

Course performance 

Percentage of core courses passed 2 

86.6 

86.6 

0.1 

0.00 

0.924 

Sample size 

29 

29 




Panel B: Middle school sample 

Attendance 

Percentage of enrolled days attended 

92.7 

91.4 

1.3 

0.12 

0.186 

Behavior 

Percentage of enrolled days suspended or expelled 

1.2 

1.2 

-0.1 

-0.01 

0.842 

Course performance 

Percentage of core courses passed 2 

94.5 

92.8 

1.7 

0.10 

0.147 

Sample size 

14 

15 




Panel C: High school sample 

Attendance 

Percentage of enrolled days attended 

86.9 

87.3 

-0.4 

-0.03 

0.704 

Behavior 

Percentage of enrolled days suspended or expelled 

1.8 

1.5 

0.3 

0.05 

0.403 

Course performance 

Percentage of core courses passed 2 

79.8 

79.2 

0.7 

0.02 

0.433 

Sample size 

15 

14 





(continued) 


38 









Table 4.1 (continued) 


SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 58 study schools, 14,950 nonrepeating sixth- and ninth-grade students are included in the 
analyses. Among the sample, 6,997 students attended DN schools and 7,953 students attended non-DN schools. 
Some students are not included in the analyses of course performance measures because data were not available on 
their grades for specific courses, and an entire DN middle school was dropped from these analyses because there 
are no baseline course data for the students attending that school. There are no more than 8 percent missing DN 
school students and 8 percent missing non-DN school students for any of the course performance measures. 

Estimated impacts are based on a two-level model with students nested within schools, controlling for random 
assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 
observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 
column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 
covariate values for students in DN schools as the basis for the adjustment. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a The denominator includes all core courses (math, English/language arts, science, or social studies) each student 
attempted during the school year. 


promising path to graduation) as well as those with no early warning indicators (students who 
are not primary targets for intervention to get them on track to graduation). As is shown in Ta- 
ble 4.2, there is one measure on which Diplomas Now had a statistically significant impact: the 
percentage of students with no early warning indicators. On average, in the DN schools, just 
over 63 percent of the sixth- and ninth-grade students met the three conditions: better than an 85 
percent attendance rate, fewer than three days of disciplinary action that kept them out of clas- 
ses, and no failed English/language arts or math courses. At the non-DN schools, 59.5 percent 
of their peers met these thresholds, resulting in an impact of 3.6 percentage points. 

Table 4.3 displays the impact findings specific to middle schools. Diplomas Now had 
statistically significant, positive impacts on the percentage of students with an attendance rate 
greater than 90 percent and also on the percentage of students with no early warning indicators. 
The impact estimates for these outcomes were 4.1 percentage points and 5.5 percentage points, 
respectively. On average, there were 1 93 sixth-grade students per school in the analysis sample; 
therefore, these impact estimates suggest that about 8 more students had better than 90 percent 
attendance and more than 10 students did not have early warning indicators, on average, in DN 
schools compared with non-DN schools. Supplemental analyses that accounted specifically for 
the grade configurations of middle schools in the sample also found positive impacts on attend- 
ance, course perfonnance, and composite on-track outcomes, further supporting the impression 
that Diplomas Now is making a difference in middle schools. In particular, these analyses 
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Table 4.2 


Impacts on Threshold Measures of 
Attendance, Behavior, and Course Performance, Cohort 2 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P- Value for 

Estimated 

Impact 

Attendance 

Percentage of students who attended over 

90 percent of enrolled days 

68.3 

67.5 

0.8 

0.02 

0.601 

Percentage of students who attended over 

85 percent of enrolled days 

79.7 

79.3 

0.4 

0.01 

0.787 

Behavior 

Percentage of students who were ever suspended 
or expelled during year 

23.8 

21.3 

2.5 

0.06 

0.415 

Percentage of students who were suspended or 
expelled for 3 or more days 

16.8 

15.2 

1.7 

0.05 

0.472 

Course performance 

Percentage of students who had no core course 
failures during year 2 

73.5 

72.8 

0.7 

0.02 

0.675 

Percentage of students who had no math course 
failures during year 

84.5 

82.1 

2.3 

0.06 

0.233 

Percentage of students who had no English/ 
language arts course failures during year 

87.5 

86.4 

1.0 

0.03 

0.382 

ABC composite measures b 

Percentage of students above stability threshold 

50.6 

48.1 

2.5 

0.05 

0.210 

Percentage of students with no early warning 

indicators 

63.2 

59.5 

3.6 

0.07 ** 

0.039 

Sample size 

29 

29 





(continued) 


accounted for whether entry into sixth grade represented a true transition experience, because 
the school’s earliest grade was sixth, or whether it represented more of a continuation, as in the 
few schools with grades 5-8 or grades K-8 configurations. (See Appendix Table B.6.) 2 


Supplemental analyses (or “sensitivity tests”) accounting for other school-level characteristics (for exam- 
ple, percentages of teachers with different certification levels) were conducted and did not reveal different pat- 
terns of outcomes from what is presented in this chapter for the main analyses. Various supplemental analyses 
are discussed in Appendix B. 
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Table 4.2 (continued) 


SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 58 study schools, 14,950 nonrepeating sixth- and ninth-grade students are included in the 
analyses. Among the sample, 6,997 students attended DN schools and 7,953 students attended non-DN schools. 
Some students are not included in the analyses of course performance measures because data were not available 
on their grades for specific courses, and an entire DN middle school was dropped from these analyses because 
there are no baseline course data for the students attending that school. There are no more than 14 percent 
missing DN school students and 9 percent missing non-DN school students for any of the course performance 
measures. The large percentage of missing DN school students is due to one school with a large amount of 
missing math course data. Without this school included, there are no more than 9 percent missing students on 
any of the measures. A sensitivity test removing the random assignment block that includes this school from the 
analyses resulted in similar findings. 

Estimated impacts are based on a two-level model with students nested within schools, controlling for 
random assignment block and school- and student-level covariates. The values in the column labeled “DN 
Schools” are the observed mean outcomes weighted by the number of schools in a random assignment block, 
and the values in the column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN 
schools using the mean covariate values for students in DN schools as the basis for the adjustment. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 
a Core courses include math. English/language arts, science, or social studies courses. 
b The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 
district, was never suspended or expelled, and did not fail any core courses attempted during the school year. 
The “no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the 
district, was suspended or expelled for fewer than three days, and did not fail any math or English/language arts 
courses during the school year. If a student has missing course data but meets the other two criteria for an ABC 
composite measure, the student is assigned a value of 1 . 

The high school findings presented in Table 4.4 are not as promising as those for the 
middle schools. None of the impact estimates presented in this table are statistically significant, 
suggesting that the implementation of Diplomas Now did not result in better outcomes for 
ninth-graders than what was happening in the non-DN schools. There are more positive esti- 
mates across the nine threshold measures for middle schools than for high schools, suggesting 
that Diplomas Now may be more effective at the middle school level than at the high school 
level. However, for only one measure — the percentage of students with better than 90 percent 
attendance — do the impacts at the middle school and high school levels differ to a statistically 
significant degree. 3 That is, Diplomas Now is having a greater impact on keeping students from 
being chronically absent in middle school than in high school. 


3 The “dagger” to the right of the p-values for the middle school and high school impact estimates on that 
measure indicates that the two impact estimates are statistically different from each other (p < 0.10). This sym- 
bol has the same meaning when used in other tables in this report. 
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Table 4.3 


Impacts on Threshold Measures of Attendance, 
Behavior, and Course Performance, Middle Schools, Cohort 2 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P-Value for 

Estimated 

Impact 

Attendance 

Percentage of students who attended over 

90 percent of enrolled days 

77.2 

73.0 

4.1 

0.09 * 

0.073 f 

Percentage of students who attended over 

85 percent of enrolled days 

86.8 

84.2 

2.5 

0.07 

0.300 

Behavior 

Percentage of students who were ever suspended 
or expelled during year 

22.1 

21.6 

0.5 

0.01 

0.932 

Percentage of students who were suspended or 
expelled for 3 or more days 

15.6 

16.1 

-0.4 

-0.01 

0.916 

Course performance 

Percentage of students who had no core course 
failures during year a 

87.0 

85.7 

1.3 

0.04 

0.631 

Percentage of students who had no math course 
failures during year 

94.3 

93.6 

0.7 

0.02 

0.753 

Percentage of students who had no English/ 
language arts course failures during year 

95.5 

93.5 

2.1 

0.10 

0.101 

ABC composite measure b 

Percentage of students above stability threshold 

62.5 

57.6 

4.8 

0.10 

0.181 

Percentage of students with no early warning 

indicators 

74.5 

69.0 

5.5 

0.12 * 

0.076 

Sample size 

14 

15 





(continued) 


Impacts for More- and Less-Prepared Students 

The Diplomas Now partnership, through implementation of its reform model, intends both to 
get students who are on less stable paths to graduation onto more stable paths and to keep stable 
students on track. The evaluation team analyzed the impact of Diplomas Now on the outcomes 
of students entering high school based on whether they met the stability thresholds in eighth 
grade (that is, whether they exceeded the more stringent threshold for the ABC outcomes in 
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Table 4.3 (continued) 


SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 29 study middle schools, 5,606 nonrepeating sixth-grade students are included in the analyses. 
Among the sample, 2,513 students attended DN schools and 3,093 students attended non-DN schools. Some 
students are not included in the analyses of course performance measures because data were not available on their 
grades for specific courses, and an entire DN middle school was dropped from these analyses because there are no 
baseline course data for the students attending that school. There are no more than 9 percent missing DN school 
students and 12 percent missing non-DN school students for any of the course performance measures. 

Estimated impacts are based on a two-level model with students nested within schools, controlling for random 
assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are 
the observed mean outcomes weighted by the number of schools in a random assignment block, and the values in 
the column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the 
mean covariate values for students in DN schools as the basis for the adjustment. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistically significant differences in impacts between the middle school and high school groups are indicated 
by t when the p-value is less than or equal to 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

“Core courses include math. English/language arts, science, or social studies courses. 
b The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 
district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 
“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, 
was suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses 
during the school year. If a student has missing course data but meets the other two criteria for an ABC composite 
measure, the student is assigned a value of 1 . 

their last year before entering high school). Because elementary school data across participating 
study districts did not consistently include the disciplinary and course performance metrics used 
for the stability indicators, the evaluation team could not do the same analysis for students enter- 
ing middle school. Instead, standardized state assessment scores were used to represent stu- 
dents’ preparedness for middle school as a proxy for being on a more stable trajectory or not, 
and the impact of Diplomas Now was analyzed according to whether students entering middle 
school had achieved academic proficiency by the end of elementary school. How did the impact 
of the Diplomas Now model differ for students entering high school or middle school based on 
their readiness for the next level of school? 

Table 4.5 presents the impacts on student outcomes for ninth-graders who were or were 
not above the stability threshold level at the end of eighth grade (see Panel A and Panel B, re- 
spectively). The table displays one statistically significant impact. Among the students who 
were above the stability threshold in eighth grade, there is an estimated 8 percentage point im- 
pact on the percentage of students who did not fail math in ninth grade. This is equivalent to 
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Table 4.4 


Impacts on Threshold Measures of Attendance, 
Behavior, and Course Performance, High Schools, Cohort 2 



DN 

Non-DN 

Estimated 

Effect 

P-Value for 

Estimated 

Outcome 

Schools 

Schools 

Impact 

Size 

Impact 

Attendance 

Percentage of students who attended over 

90 percent of enrolled days 

60.1 

61.9 

-1.8 

-0.04 

0.472 f 

Percentage of students who attended over 

85 percent of enrolled days 

73.1 

74.5 

-1.4 

-0.04 

0.383 

Behavior 

Percentage of students who were ever suspended 
or expelled during year 

25.4 

21.1 

4.3 

0.12 

0.167 

Percentage of students who were suspended or 
expelled for 3 or more days 

18.0 

15.4 

2.6 

0.08 

0.368 

Course performance 

Percentage of students who had no core course 
failures during year 3 

61.8 

60.8 

1.0 

0.02 

0.531 

Percentage of students who had no math course 
failures during year 

75.9 

74.1 

1.8 

0.04 

0.603 

Percentage of students who had no English/ 
language arts course failures during year 

80.5 

78.9 

1.6 

0.04 

0.486 

ABC composite measure b 

Percentage of students above stability threshold 

39.6 

38.9 

0.7 

0.01 

0.726 

Percentage of students with no early warning 

indicators 

52.6 

50.0 

2.6 

0.05 

0.257 

Sample size 

15 

14 





(continued) 


about 26 more stable entering ninth-graders not failing math in DN schools compared with non- 
DN schools. 

Overall, the pattern of results looks different for the two subgroups of students, with the 
results appearing more positive in general for the stable entering ninth-grade students. On one 
measure — the percentage of students at stability levels in ninth grade — the difference in im- 
pacts between the two groups of students is statistically significant: Among the students who 
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Table 4.4 (continued) 


SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 29 study high schools, 9,344 nonrepeating ninth-grade students are included in the analyses. 
Among the sample, 4,484 students attended DN schools and 4,860 students attended non-DN schools. Some 
students are not included in the analyses of course performance measures because data were not available on their 
grades for specific courses. There are no more than 16 percent missing DN school students and 9 percent missing 
non-DN school students for any of the course performance measures. The large percentage of missing DN school 
students is due to one school with a large amount of missing math course data. Without this school included, there 
are no more than 9 percent missing students on any of the measures. A sensitivity test removing the random 
assignment block that includes this school from the analyses resulted in similar findings. 

Estimated impacts are based on a two-level model with students nested within schools, controlling for random 
assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are 
the observed mean outcomes weighted by the number of schools in a random assignment block, and the values in 
the column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the 
mean co variate values for students in DN schools as the basis for the adjustment. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistically significant differences in impacts between the middle school and high school groups are indicated 
by f when the p-value is less than or equal to 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Core courses include math. English/language arts, science, or social studies courses. 

b The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 
district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 
"no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, 
was suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses 
during the school year. If a student has missing course data but meets the other two criteria for an ABC composite 
measure, the student is assigned a value of 1 . 


came into high school on a stable path, Diplomas Now had a 5 percentage point impact on the 
percentage who were still maintaining stability levels at the end of the year, compared with a 
negative 1.8 percentage point impact among those who had not been on a more stable path in 
eighth grade. This indicates that, in high school, the Diplomas Now model was more effective 
at keeping students on a stable path than getting students on a stable path. 

The evaluation team also analyzed the impact of Diplomas Now for two subgroups of 
middle school students: one group of students who were rated proficient or better on both their 
math and English/language arts state assessments in the prior year, and another group of stu- 
dents rated less than proficient on at least one of those exams. In general, the pattern of impacts 
is similar across both groups, and none of the impacts differ by a statistically significant amount 
between the groups. Of note is that there are relatively large positive impacts for both groups for 
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Table 4.5 


Impacts on Threshold Measures of 
Attendance, Behavior, and Course Performance, 
Students Above and Below Stability Threshold in Previous Year, 
High Schools, Cohort 2 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P- Value for 

Estimated 

Impact 

Panel A: Students above stability threshold in 

previous vear 

Attendance 

Percentage of students who attended over 

90 percent of enrolled days 

80.3 

79.3 

1.0 

0.03 

0.731 

Percentage of students who attended over 

85 percent of enrolled days 

90.7 

90.4 

0.4 

0.02 

0.898 

Behavior 

Percentage of students who were ever suspended 
or expelled during year 

13.7 

13.3 

0.4 

0.01 

0.893 

Percentage of students who were suspended or 
expelled for 3 or more days 

8.6 

5.9 

2.6 

0.13 

0.302 

Course performance 

Percentage of students who had no core course 
failures during year 2 

80.4 

76.7 

3.7 

0.08 

0.172 

Percentage of students who had no math course 
failures during year 

90.2 

81.9 

8.2 

0.22 

** 0.018 

Percentage of students who had no English/ 
language arts course failures during year 

92.1 

90.0 

2.1 

0.07 

0.327 

ABC composite measure b 

Percentage of students above stability threshold 

62.7 

57.7 

5.0 

0.10 

0.148 f 

Percentage of students with no early warning 

indicators 

76.4 

71.7 

4.7 

0.11 

0.193 

Sample size 

15 

14 





(continued) 
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Table 4.5 (continued) 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P- Value for 

Estimated 

Impact 

Panel B: Students below stability threshold in 

previous year 

Attendance 

Percentage of students who attended over 

90 percent of enrolled days 

41.3 

45.0 

-3.6 

-0.07 

0.212 

Percentage of students who attended over 

85 percent of enrolled days 

57.9 

59.4 

-1.6 

-0.03 

0.575 

Behavior 

Percentage of students who were ever suspended 
or expelled during year 

38.8 

30.9 

7.9 

0.17 

0.144 

Percentage of students who were suspended or 
expelled for 3 or more days 

28.9 

24.6 

4.3 

0.10 

0.359 

Course performance 

Percentage of students who had no core course 
failures during year 2 

44.3 

43.6 

0.7 

0.01 

0.760 

Percentage of students who had no math course 
failures during year 

62.3 

62.2 

0.1 

0.00 

0.986 

Percentage of students who had no English/ 
language arts course failures during year 

69.3 

64.9 

4.4 

0.09 

0.219 

ABC composite measure* 5 

Percentage of students above stability threshold 

18.4 

20.2 

-1.8 

-0.04 

0.320 f 

Percentage of students with no early warning 

indicators 

32.1 

29.5 

2.6 

0.06 

0.291 

Sample size 

15 

14 





(continued) 


the stability and early warning threshold measures of attendance and being on track. For the 
nonproficient group, the 4.6 percentage point impact estimate on the 90 percent attendance sta- 
bility threshold and the 7 percentage point impact on having no early warning indicators were 
statistically significant. These findings are presented in Appendix Table B. 1 . 
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Table 4.5 (continued) 


SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 29 study high schools, 7,500 nonrepeating ninth-grade students are included in the analyses. 
Among the sample, 3,577 students attended DN schools and 3,923 students attended non-DN schools. The above- 
threshold subgroup includes 4,328 students and the below-threshold subgroup includes 3,172 students. Some 
students are not included in the analyses of course performance measures because data were not available on their 
grades for specific courses. There are no more than 20 percent missing DN school students and 1 1 percent 
missing non-DN school students for any of the course performance measures. The large percentage of missing 
DN school students is due to one school with a large amount of missing math course data. Without this school 
included, there are no more than 1 1 percent missing students on any of the measures. A sensitivity test removing 
the random assignment block that includes this school from the analyses resulted in similar findings. 

Estimated impacts are based on a two-level model with students nested within schools, controlling for random 
assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are 
the observed mean outcomes weighted by the number of schools in a random assignment block, and the values in 
the column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the 
mean covariate values for students in DN schools as the basis for the adjustment. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistically significant differences in impacts between the groups above and below the stability threshold in 
the previous year are indicated by f when the p-value is less than or equal to 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Core courses include math. English/language arts, science, or social studies courses. 

b The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 
district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 
“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, 
was suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses 
during the school year. If a student has missing course data but meets the other two criteria for an ABC composite 
measure, the student is assigned a value of 1 . 


Impacts on ABC Outcomes During the First Year of 
Implementation 

Were impacts different for the first cohorts of sixth- and ninth-grade students? Table 4.6 dis- 
plays the impacts of the Diplomas Now model during the first year of implementation on stu- 
dents’ ABC outcomes — the impacts for the first cohort in each school. As with the second co- 
hort of students, there was one statistically significant impact on the threshold outcomes during 
the first year of implementation. Diplomas Now has an almost 4 percentage point impact on the 
percentage of students who did not fail math in sixth or ninth grade. The estimated impact on 
the percentage of first-cohort students above the ABC composite stability threshold at the end of 
their sixth- and ninth-grade years is negative but not statistically significant. Still, there is a sta- 
tistically significant difference between the first and second years of implementation on this 
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Table 4.6 


Impacts on Threshold Measures of 
Attendance, Behavior, and Course Performance, Cohort 1 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P- Value for 

Estimated 

Impact 

Attendance 

Percentage of students who attended over 

90 percent of enrolled days 

66.8 

68.2 

-1.4 

-0.03 

0.418 

Percentage of students who attended over 

85 percent of enrolled days 

78.2 

78.9 

-0.8 

-0.02 

0.650 

Behavior 

Percentage of students who were ever suspended 
or expelled during year 

25.7 

22.3 

3.4 

0.09 

0.276 

Percentage of students who were suspended or 
expelled for 3 or more days 

18.0 

16.4 

1.7 

0.05 

0.501 

Course performance 

Percentage of students who had no core course 
failures during year 3 

73.2 

70.9 

2.3 

0.05 

0.318 

Percentage of students who had no math course 
failures during year 

86.0 

82.1 

3.9 

0.10 ** 

0.029 

Percentage of students who had no English/ 
language arts course failures during year 

86.4 

86.1 

0.4 

0.01 

0.850 

ABC composite measure 13 

Percentage of students above stability threshold 

47.3 

49.9 

-2.6 

-0.05 

0.164 f 

Percentage of students with no early warning 

indicators 

60.9 

60.7 

0.2 

0.00 

0.894 

Sample size 

31 

29 





(continued) 


measure, as represented by the dagger in Table 4.6. Although the difference between DN and 
non-DN schools in the second year was not statistically significant, the effect the Diplomas 
Now model had on this measure had improved by a statistically significant amount from the 
prior year. This may have been driven predominantly by sixth-graders, as there was a similar 
statistically significant improvement in the percentage of students on track in the middle 
schools. (See Panel A of Appendix Table B.3.) 
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Table 4.6 (continued) 


SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 60 study schools, 15,280 nonrepeating sixth- and ninth-grade students are included in the analyses. 
Among the sample, 7,322 students attended DN schools and 7,958 students attended non-DN schools. Some 
students are not included in the analyses of course performance measures because data were not available on their 
grades for specific courses, and an entire DN middle school was dropped from these analyses because there are no 
baseline course data for the students attending that school. There are no more than 9 percent missing DN school 
students and 10 percent missing non-DN school students for any of the course performance measures. 

Estimated impacts are based on a two-level model with students nested within schools, controlling for random 
assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 
observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 
column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 
covariate values for students in DN schools as the basis for the adjustment. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistically significant differences in impacts between the Cohort 1 and Cohort 2 groups are indicated by f 
when the p-value is less than or equal to 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Core courses include math. English/language arts, science, or social studies courses. 

b The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 
district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 
"no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, was 
suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 
the school year. If a student has missing course data but meets the other two criteria for an ABC composite 
measure, the student is assigned a value of 1 . 

There was a small increase in the average percentage of students on a stable path at DN 
schools between the first and second year. As shown in Table 4.6, on average at DN schools, 
47.3 percent of the first-cohort students were achieving the composite stability threshold at the 
end of the first year of implementation, while in the second year of implementation, 50.6 per- 
cent of the second-cohort students achieved that threshold (see Table 4.2). At the non-DN 
schools, there was a slight decline in the percentage of stable students from Year 1 to Year 2, 
from 49.9 percent to 48.1 percent. 4 This increase in DN schools and decrease in non-DN 
schools mirrors some of the service contrast findings that grew between the first and second 
years, owing, in part, to a decline in the non-DN schools’ average service levels, as discussed in 


4 Although the impact estimates for the first and second cohorts of students on the lower threshold on-track 
indicator (that is, the percentage of students with no early warning indicators) do not differ to a statistically 
significant degree, there is a similar pattern, in which the percentage of students exceeding the lower threshold 
increases in the DN schools and decreases in the non-DN schools. 
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Chapter l. 5 Although there are only two time points, this growth in the percentage of students 
on track from the first year to the second year offers some promise for future growth in impacts 
as the DN school staff members build their understanding and implementation of the model. 


Variation in Impacts Across Schools 

Table 4.2 shows the average impact of the Diplomas Now model on the percentage of students 
on a stable path during the second year of implementation. As discussed earlier in this chapter, 
on average across all the blocks of DN and non-DN schools in the model, the Diplomas Now 
model does not show a statistically significant impact on the percentage of students above the 
composite stability threshold. Still, it is possible that for some DN schools the model could be 
generating more of an impact than in others. Table 4.2 also shows a statistically significant im- 
pact on the percentage of students with no early warning indicators, the less stringent threshold 
for the ABC composite measure. It is possible that the overall impact on this measure is driven 
more by implementation in some DN schools compared with others. 

To investigate this possibility, Figure 4.1 and Figure 4.2 display the impact estimates 
for each of the random assignment blocks in the study for the upper and lower threshold on- 
track indicators. As discussed in Chapter 2, random assignment blocks are a set of like 
schools within a school district that were randomly assigned at the same time to implement 
Diplomas Now or not to implement Diplomas Now. The blocks range from two to six 
schools, and each block has both DN and non-DN schools, but only middle schools or only 
high schools. There are 21 blocks in the second-year analytic sample. In Figure 4.1, statisti- 
cally significant differences in the percentage of stable students in DN schools compared with 
non-DN schools are indicated by vertical bars that do not cross the 0 line. As the figure indi- 
cates, only two of the random assignment blocks show statistically significant, positive mean 
differences, even though over half the blocks had positive mean differences (shown by the 
circle or square being above the 0 line). In Figure 4.2, three of the random assignment blocks 
show statistically significant, positive mean differences on the percentage of students with no 
early warning indicators while over half the blocks had positive mean differences. Although 
there is some variation in the impacts across the blocks in each of the figures, a test across all 
blocks indicates that this variation across schools is not statistically significant. This suggests 
that there was not a lot of variation in DN schools’ ability to effect change, and while more 
blocks show a positive difference for DN schools compared with their non-DN counterparts, 
this variation is small and may be due to chance. 


5 For in-depth comparison of service contrast between Year 1 and Year 2, see Sepanik et al. (2015), pp. 
48-54. 
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Estimated impact (percentage points) 


Figure 4.1 

Impacts on Percentage of Students Above Stability Threshold, 
by Random Assignment Block, Cohort 2 



(continued) 




Figure 4.1 (continued) 


SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 58 study schools, there are 21 random assignment (RA) blocks. 

These fixed-effects impact estimates are based on a two-level model with students nested within 
schools controlling for random assignment block and school- and student-level covariates. The vertical 
error bars represent a 90 percent confidence interval for each data point. 

A composite F-test was used to assess whether the variation in impacts across random assignment 
blocks is larger than would be expected due to chance. The result is not statistically significant. 

The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in 
the district, was never suspended or expelled, and did not fail any core courses attempted during the 
school year. If a student has missing course data but meets the other two criteria for an ABC composite 
measure, the student is assigned a value of 1 . 


Conclusion 

In the second year of its implementation under i3, the Diplomas Now model did not have statis- 
tically significant impacts on sixth- and ninth-grade students’ average attendance rates, days 
seriously disciplined, or course passing rates. In terms of the composite measures of attendance, 
behavior, and course performance there was no statistically significant impact on the percentage 
of stable students, but there was a small statistically significant impact on the percentage of stu- 
dents with no early warning indicators. However, the impact story becomes more nuanced when 
one looks specifically at middle schools or high schools. There are signs that implementation of 
the Diplomas Now model may have led to some improvement on the ABC outcomes of middle 
school students, particularly regarding attendance and having no early warning indicators, and 
the effect was similar for students regardless of their academic proficiency level coming into 
middle school. At the high school level, the Diplomas Now model appears not to have much 
impact compared with what was happening in the non-DN schools. Within high schools, how- 
ever, there appears to be some differentiation in how well Diplomas Now may be meeting the 
needs of students coming into high school who are on more or less stable paths to graduation. It 
does appear that the implementation of the Diplomas Now model has helped stable students 
stay on that trajectory more than it has helped other students get onto that trajectory. 

Although the Diplomas Now model did not have an overall impact on most of the ABC 
outcomes measured by this study during the second year of implementation, the program is 
complex and could take more than two years to fully take hold in a school in a way that leads to 
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Estimated impact (percentage points) 


Figure 4.2 

Impacts on Percentage of Students with No Early Warning Indicators, 
by Random Assignment Block, Cohort 2 
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Figure 4.2 (continued) 

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 58 study schools, there are 21 random assignment (RA) blocks. 

These fixed-effects impact estimates are based on a two-level model with students nested within 
schools controlling for random assignment block and school- and student-level covariates. The vertical 
error bars represent a 90 percent confidence interval for each data point. 

A composite F-test was used to assess whether the variation in impacts across random assignment 
blocks is larger than would be expected due to chance. The result is not statistically significant. 

The “no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled 
in the district, was suspended or expelled for fewer than three days, and did not fail any math or 
English/language arts courses during the school year. If a student has missing course data but meets 
the other two criteria for an ABC composite measure, the student is assigned a value of 1. 


impacts on student outcomes. 6 The impact on the percentage of students with no early warning 
indicators (see Table 4.2) suggests that Diplomas Now is positively affecting the percentage of 
students meeting incremental thresholds that align with outcome levels targeted by the Diplo- 
mas Now staff members. Still, the model did not show a statistically significant impact on stu- 
dents meeting any of the separate early warning indicator thresholds specific to attendance, be- 
havior, or course perfonnance. Few of the early school and student outcomes measured show a 
difference between DN and non-DN schools after the first year, but it is possible that these hy- 
pothesized “early” outcomes could take longer to be fully realized since changing school cli- 
mate and student attitudes and behaviors is not generally a short-term endeavor. The one early 
student outcome with a positive impact is directly related to the model components that in- 
volved the placement of Diplomas Now staff in schools — increasing the number of adults in 
the building — and that involved some of the specific activities these staff members supported. 
This outcome suggests that the model is causing some change in students’ experiences even 
after just the first year. Furthermore, the impact of the Diplomas Now model on the percentage 
of students on a stable path to graduation was stronger in the second than in the first year, sug- 
gesting that although not yet statistically significant after two years, the program effects may be 
growing. But based on data for only two years, it is still too early to know for sure. 

The ultimate goal of the Diplomas Now model is to help more high school students 
graduate and more middle school students successfully complete ninth grade on time. Other 
than the estimated impact on the percentage of students with no early warning indicators, there 
are no statistically significant impacts on the intermediate ABC outcomes hypothesized to lead 
to these long-term goals, but the exploratory analyses in this chapter offer some tentative signs 


6 For example, Borman, Hewes, Overman, and Brown (2003) conducted a meta-analysis of whole-school 
reform models and found that those implemented for five years or more had greater impacts than those imple- 
mented for fewer years. 
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that impacts on longer-term outcomes could emerge, particularly for the middle school students. 
If affecting students around the early warning indicator level is most important, then having a 
positive impact on the lower threshold ABC composite measure may bode well for keeping or 
getting more students on track through graduation and successful ninth-grade completion. 7 
Whereas if having impacts on specific, individual ABC outcomes ends up being a more im- 
portant predictor of later success, impacts on longer-term outcomes may be less likely. 8 It is also 
possible that the model takes more than two years to fully develop the capability to cause posi- 
tive change at schools, and the growth between the first and second cohorts, if part of a trend, 
could lead to more positive ABC impacts in future years. 

The next report will investigate relationships between implementation and outcome 
data. A final report will present the same ABC outcomes as this report but for the fourth cohort 
of sixth- and ninth-grade students, answering the question of whether Diplomas Now could be 
more effective for entering students after two more years of implementation in the schools. It 
will also look at the four-year impacts of Diplomas Now on Cohort 1 and Cohort 2 high school 
students’ graduation and middle school students’ completion of ninth grade. 


7 Frazelle and Nagel (2015, pp. 7-8) give examples of multi-indicator Early Warning Systems from the 
National High School Center and Johns Hopkins University. 

8 For example, on-track indicators used in Chicago and New York City are based on course and test per- 
formance and do not include attendance and disciplinary outcomes in their definitions. Allensworth and Easton 
(2005) from the Consortium on Chicago School Research define ninth-graders as being on track to graduation 
based solely on course -related outcomes: earning at least five course credits and no more than one semester F 
in a core course. Kemple, Segeritz, and Stephenson (2013) from the Research Alliance for New York City 
Schools discuss the New York City Department of Education’s on-track measure, which is similar to Chica- 
go’s and uses credit earning to define being on track; they also discuss how passing at least one Regents exam 
(a subject area assessment administered in New York State) in ninth grade improves the predictability of 
whether a New York City ninth-grader will earn a Regents high school diploma. 
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Appendix A 

Samples, Analytic Methods, and 
Early Outcome Measures 




Chapter 2 summarizes the data, measures, samples, and analyses used in this report. This 
appendix offers additional information in each of these areas, including data sources, response 
rates, and sample construction for the ABC outcomes discussed in Chapter 4 and the early 
school and student outcomes discussed in Chapter 3; the analytic model used for all the anal- 
yses; the approximate statistical power for detecting effects on the ABC outcomes; and a 
description of the construction of the early school and student outcome measures using adminis- 
trator, teacher, and student survey items. 


Data Sources, Response Rates, and Samples for ABC Outcomes 

As discussed in Chapter 2, the Diplomas Now intervention uses a school-level random assign- 
ment design, but student-level data are used in the analyses of the ABC outcomes. All data used 
for the impact analyses discussed in Chapter 4 and Appendix B were obtained from individual 
administrative student records collected from each of the 1 1 participating school districts, as 
described in Chapter 2. Student administrative records data were collected for students who 
were enrolled in grades six and nine in the study schools during the first and second years of 
Diplomas Now implementation (201 1-2012 and 2012-2013 for the schools recruited in the first 
wave and 2012-2013 and 2013-2014 for schools recruited in the second wave). For these two 
cohorts of students, data were also collected from the year before student enrollment in study 
schools (fifth-grade data for middle school students and eighth-grade data for high school 
students) for use in the baseline analyses of state test proficiency found in Table 2.2 and as 
covariates in the impact analyses. In addition, administrative student records data were collected 
on sixth- and ninth-grade students who attended the study schools the year before implementa- 
tion (2010-201 1 for wave 1 schools and 201 1-2012 for wave 2 schools). These data were used 
to compare DN and non-DN schools at baseline on the outcome measures to confirm that 
random assigmnent produced comparable groups. The findings from this analysis are shown in 
Table 2.1. Aggregate measures of the ABC outcomes at baseline were also created from these 
data and used as covariates in the impact estimation models. 

Sixty-two schools, including 29 high schools and 33 middle schools, were randomly as- 
signed within blocks either to implement the Diplomas Now model (DN schools) or not to 
implement the Diplomas Now model (non-DN schools). The schools not assigned to implement 
the model could continue their programming or implement any other reforms. Over the first two 
years of program implementation, a few schools were dropped from the study because of school 
closures or differences in grade configurations between DN and non-DN schools in the same 
random assigmnent block. Appendix Table A.l presents the number of schools in each ABC 
outcome analysis and indicates what percentage of schools originally randomly assigned that 
number represents. Panel A presents these school-level response rates for the main analysis 
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Appendix Table A.l 


Number and Percentage of Randomly Assigned Schools in Each Analysis Sample 



DN Schools 

Non-DN Schools 

All Schools 

Outcome Measure 

Analysis 

Sample 

Size 

Percentage 

of RA 

Schools 3 

Analysis 

Sample 

Size 

Percentage 

of RA 

Schools 3 

Analysis 

Sample 

Size 

Percentage 

of RA 

Schools 3 

Panel A: Cohort 2 

Attendance, behavior, and stability 

threshold 

All schools 

29 

90.6 

29 

96.7 

58 

93.5 

Middle schools 

14 

82.4 

15 

93.8 

29 

87.9 

High schools 

15 

100.0 

14 

100.0 

29 

100.0 

Course performance 

All schools 

28 

87.5 

29 

96.7 

57 

91.9 

Middle schools 

13 

76.5 

15 

93.8 

28 

84.8 

High schools 

15 

100.0 

14 

100.0 

29 

100.0 

Panel B: Cohort 1 

Attendance, behavior, and stability 

threshold 

All schools 

31 

96.9 

29 

96.7 

60 

96.8 

Course performance 

All schools 

30 

93.8 

29 

96.7 

59 

95.2 


SOURCE: MDRC calculations based on records obtained from school districts (2012, 2013, and 2014). 

NOTES: One DN middle school and one non-DN middle school were removed from both years of analyses due to 
differences in grade configurations within the same random assignment block. One DN middle school closed prior 
to the second year of implementation, and one DN middle school was not included in the second-year analyses 
because it no longer served sixth grade. In addition, one DN middle school was not included in the course 
analyses for either year because the school did not provide baseline course data. 

“Values in column represent the percentage of randomly assigned (RA) schools used in the analysis sample for 
a given outcome measure. 


sample, representing the second cohort of sixth- and ninth-grade students to attend the study 
schools, while Panel B presents the school-level response rates for the first cohort. 

Specifically, two schools were dropped from all analyses, two were dropped from the 
second year of analyses, and one school was excluded from all course analyses, while one 
random assigmnent block used first-year data: 
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• Two middle schools from the same random assignment block, one DN school and 
one non-DN school, were not included in any analyses because one school served 
only seventh and eighth grade while the other school served sixth through eighth 
grade. The study focuses on the transition year (the first grade level that students at- 
tend the school), and in these two schools, the transition years were not comparable. 

• Two middle schools were not included in the analyses of the second cohort: One 
DN school closed before the second year of implementation, and another DN 
school stopped serving sixth grade before the second year of implementation, so it 
was no longer comparable to the other schools in the random assignment block that 
kept serving sixth grade. 

• In both the first and second cohorts, one DN middle school was dropped from all 
course perfonnance analyses because baseline course performance data, which 
were used to create a school-level covariate included in the course perfonnance 
analyses, were not available for that school. 

• In one random assignment block with two DN schools and one non-DN school, the 
two DN schools stopped serving sixth grade after the first year of implementation. 
In order to keep the block in the analyses, the Cohort 1 data for all three schools are 
used in the impact analyses for Cohort 2. 

The data received from the districts and used in the analyses represent the study team’s 
best knowledge of the full sample of students present at the schools. Only nonrepeating sixth- 
and ninth-grade students are included in the analyses because the focus of the Diplomas Now 
model is on incoming students during their transition year. There are 14,950 nonrepeating sixth- 
and ninth-grade students in Cohort 2. Among the sample, 6,997 students attended DN schools 
and 7,953 students attended non-DN schools. Students were included in the study regardless of 
how long they attended the study school or other schools in the district, but some students did 
not have course perfonnance data because they did not attend the school long enough to have 
received course grades. A total of 652 DN school students and 62 1 non-DN school students in 
Cohort 2 were dropped from the course perfomiance analyses because of missing course 
perfonnance data (including the students in the aforementioned school that was dropped from 
the course perfonnance analyses due to missing baseline data). 


Data Sources, Response Rates, and Samples for 
Early School and Student Outcomes 

As discussed in Chapter 2, data from administrator, teacher, and student surveys were used in 
the analyses of the early school and student outcome measures. Teacher and administrator 
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surveys were administered online or in pencil-and-paper format during the spring of the second 
year of implementation (2013 for wave 1 schools and 2014 for wave 2 schools). Student 
surveys were administrated using pencil-and-paper format during the spring of the first year of 
implementation (2012 for wave 1 schools and 2013 for wave 2 schools). 1 Copies of the teacher, 
administrator, and student surveys can be found in the Survey Instrument Supplement to this 
report. 2 All schools participated in the student survey conducted in the spring of the first year of 
implementation and are included in the analyses of student survey data. In schools that did not 
have sixth grade, seventh-grade student data were collected and included in the analyses. As 
noted previously, by the second year of implementation, one DN school had closed, so it is not 
included in the analyses of teacher and administrator early school outcomes, and one school was 
dropped from the teacher survey analyses because it no longer included a sixth grade in the 
second year of implementation. 

Appendix Table A.2 describes the individual-level response rates and sample sizes for 
the teacher, administrator, and student surveys used in analyses of early school and student 
outcomes. The sample sizes represent the total number of teachers, administrators, and sixth- 
and ninth-grade students at the schools, and the response rates are the percentages of those 
groups that participated in the surveys. 

The construction of the analytic samples for the early outcome analyses from the pool 
of eligible administrators and teachers is presented in Appendix Figures A.l and A.2, respec- 
tively. Appendix Figure A. 1 illustrates that 217 eligible administrators (principals and assistant 
principals) across the study schools were identified during the second year of implementation. 
Of these eligible administrators, 182 (84 percent) responded to the survey questionnaire. 
Although the study team fried to identify only principals and assistant principals to participate in 
the survey, 12 respondents listed positions other than principal or assistant principal or similar 
administrative positions. These 12 respondents were dropped from the analysis. 

Appendix Figure A.2 shows that 3,128 eligible teachers were identified during the sec- 
ond year of implementation. Of these eligible teachers, 2,418 (77 percent) responded to the 
survey questionnaire. Again, the study team tried to identify only teachers to participate in the 
survey, but 130 respondents listed their position as something other than teacher (for example, 
counselor or academic adviser). These respondents were dropped from the analyses. Since the 
focus of this report is on the impact during the transition year, the primary analysis sample is 


1 Survey data files and administrative records data are not linked at the individual student level because 
research approval policies in many participating districts regarding parent and student consent for survey 
participation and data collection led the evaluation team to collect anonymous survey data. 

2 Corrin, Sepanik, Rosen, and Shane (2016), available on MDRC’s website (www.mdrc.org). 
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Appendix Table A.2 


Individual-Level Response Rates for 
Teacher, Administrator, and Student Surveys 



DN 

Non-DN 

All 


Schools 

Schools 

Schools 

Panel A: Teachers 




Y ear 2 response rate (%) 

75.6 

78.8 

77.3 

Sample size 

1,476 

1,652 

3,128 

Panel B: Administrators 




Year 2 response rate (%) 

84.5 

83.2 

83.9 

Sample size 

110 

107 

217 

Panel C: Students 




Y ear 1 response rate (%) 

68.7 

63.0 

65.6 

Sample size 

5,935 

7,168 

13,103 


SOURCE: Survey administration documentation (2012, 2013, 2014). 

NOTES: One DN middle school closed prior to the second year of implementation and is not included 
in the analyses of teacher and administrator surveys. One DN middle school was not included in the 
analyses of teacher surveys because it no longer served sixth grade during the second year of 
implementation. All schools participated in the student survey during the first year of implementation. 


composed of the 1,339 teachers who taught sixth or ninth grade. (As noted previously, in 
schools where there was no sixth grade, seventh-grade teachers were included.) Teachers who 
did not report teaching these grades were not included in the analyses. 


Analytic Models 

Impact estimates for the ABC outcomes were obtained by using a two-level fixed-effects 
model, which accounts for the nesting of students in schools and districts. Level 1 specifies the 
relationship between student - level characteristics and outcomes for students within schools. 
This model is given by: 

Yij ~ n ()j X TCls^sij + + Cij, (1) 


where 
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Appendix Figure A.l 


Construction of the Sample of Administrators 
for the Analyses of Early School Outcomes 
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Y t j is an outcome for student i in school /; 

is a set of A* student-level covariates for student i in school j, measured before 
students’ first exposure to the Diplomas Now program; 

M si j is a set of S missing indicators for each of the student-level characteristics, 
coded 1 for students missing data on that characteristic and 0 otherwise; and 

e L j is the error term for student i from school j, assumed to be independently and 
identically distributed across students within schools. 

Therefore, 

n 0 j is the average student outcome Y at school j in the sample. 

Level 2 focuses on the variation in outcomes between schools. Because random as- 
signment occurred at the school level, the treatment indicator is a level 2 variable that estimates 
the treatment effect (that is, the impact of Diplomas Now) at the school level. Level 2 also 
includes block-level fixed effects that serve to control for the variation in estimates between 
schools and school levels (middle school or high school) in the same district. The school-level 
equation is given as: 
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Appendix Figure A.2 


Construction of the Sample of Teachers for the Analyses of Early School Outcomes 
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n 0i — YiK^kDic + YiKPlkTjDk + Em Pl-mWmj + u 00 (2) 

where 


Tj is equal to 1 if school j was randomly assigned to implement the Diplomas Now 
program and 0 otherwise; 

D k denotes random assignment block indicators equal to 1 if school j is in random 
assignment block k and 0 otherwise; 

W m j is a set of M school-level covariates for school j measured before the first year 
of program implementation; and 

u oi is a random error term for school j, assumed to be independently and identically 
distributed across schools. 

Therefore, 

1 3 lk is the difference between the school-level average of outcome Y in the DN 
schools and the non-DN schools in block/district k, 

and the two-level model can be estimated by substituting equation (2) into equation (1), and 
then fitting equation (3): 

Y ij - Elf^/A + 'Z.KPlk'TjDk + Em Plm^mj + Es^uAi; + u 0i + e ij ■ (3) 
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With these models, the average impact of the Diplomas Now program across random 
assignment blocks (/? a ) is obtained by weighting the block-level impacts (/? (fe ) by the number of 
DN schools in the block. Thus, is a fixed-effects estimate of the impact of the Diplomas Now 
program for the average DN school in the study sample. Therefore, the average estimate cannot 
be used to make statistical inferences about the impact of the program in some larger population 
of schools. This fixed-effects approach to obtaining a pooled impact estimate is used because 
the school districts in the study were selected purposefully and are not a random sample of 
districts from a larger target population. In addition, the model includes student-level covariates 
(Xjj) to reduce both within- and between-school variation in the outcome measure, which 
increases the precision of the impact estimates. The student-level covariates include race, 
gender, whether a student is overage for grade, special education status, English language 
learner status, previous year math and English state standardized test proficiency level, and a 
previous year version of the outcome measure when it was available. 3 For each outcome, one 
school-level covariate (fV m j) — a baseline measure of the outcome — was also included. It 
reduces between-school variation in outcomes and increases the precision of the impact 
estimates. 

Similar two-level fixed-effects models were also used to obtain the difference estimates 
for the early school and student outcomes reported in Chapter 3, accounting for nesting of 
students, teachers, or administrators within schools and districts. Student, teacher, and adminis- 
trator characteristics were included as covariates at the individual level. These covariates were 
all created from survey items, and they include students’ race and gender and whether a student 
is overage for grade; teachers’ degree level, certification status, and years of experience; and 
administrators’ years of experience. No school-level covariates were included in the analyses of 
early school and student outcomes. 


Statistical Power for Analyses of ABC Outcomes 

The goal of this evaluation was to be able to detect impacts on student ABC outcomes that are 
policy relevant. To do this the study team had to ensure that a minimum number of schools 
were enrolled in the study and randomly assigned. The original goal was to recruit a total of 80 
schools, 40 middle schools and 40 high schools. In the end, the program was able to recruit 62 
schools; 58 of those schools are included in the Cohort 2 analyses (29 middle schools and 29 
high schools). Appendix Table A.3 reports the approximate minimum detectable effect (MDE) 


3 Fifth-grade data do not generally include suspension/expulsion data or course data. Those measures are 
not included as covariates in the main analyses or in the middle school subgroup analyses but are included 
as covariates in the high school subgroup analyses. 
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Appendix Table A.3 


Minimum Detectable Effect and Effect Size for Impacts on 
ABC Outcomes, Cohort 2 


Outcome 

Number of 

Schools 

MDE 

MDES 

Panel A: All schools 

Attendance 

Percentage of enrolled days attended 

58 

1.76 

0.13 

Percentage of students who attended over 

90 percent of enrolled days 

58 

4.03 

0.09 

Percentage of students who attended over 

85 percent of enrolled days 

58 

3.40 

0.09 

Behavior 

Percentage of enrolled days suspended or expelled 

58 

0.55 

0.10 

Percentage of students who were ever suspended 
or expelled during year 

58 

7.81 

0.20 

Percentage of students who were suspended or 
expelled for 3 or more days 

58 

5.95 

0.18 

Course performance 3 

Percentage of core courses passed 

57 

2.20 

0.09 

Percentage of students who had no core course 
failures during year 

57 

4.44 

0.10 

Percentage of students who had no math course 
failures during year 

57 

4.91 

0.13 

Percentage of students who had no English/ 
language arts course failures during year 

57 

3.04 

0.09 

ABC composite measure* 5 

Percentage of students above stability threshold 

58 

5.02 

0.10 

Percentage of students with no early warning 
indicators 

58 

4.20 

0.09 




(continued) 
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Appendix Table A.3 (continued) 


Outcome 

Number of 

Schools 

MDE 

MDES 

Panel B: Middle schools 

Attendance 

Percentage of enrolled days attended 

29 

2.55 

0.22 

Percentage of students who attended over 

90 percent of enrolled days 

29 

5.48 

0.13 

Percentage of students who attended over 

85 percent of enrolled days 

29 

6.27 

0.18 

Behavior 

Percentage of enrolled days suspended or expelled 

29 

0.70 

0.13 

Percentage of students who were ever suspended 
or expelled during year 

29 

15.08 

0.35 

Percentage of students who were suspended or 
expelled for 3 or more days 

29 

10.40 

0.28 

Course performance 3 

Percentage of core courses passed 

28 

2.95 

0.17 

Percentage of students who had no core course 
failures during year 

28 

7.17 

0.21 

Percentage of students who had no math course 
failures during year 

28 

5.61 

0.21 

Percentage of students who had no English/ 
language arts course failures during year 

28 

3.04 

0.14 

ABC composite measure b 

Percentage of students above stability threshold 

29 

9.06 

0.18 

Percentage of students with no early warning 
indicators 

29 

7.44 

0.16 


(continued) 


and minimum detectable effect size (MDES) for the ABC outcome analyses for Cohort 2 across 
all schools and for middle schools and high schools separately. Intuitively, the MDE is the 
smallest program impact that could be estimated with confidence, given random sampling and 
estimation error. 4 As can be seen in Appendix Table A.3, the study is able to detect an impact of 


4 An MDE is defined as the smallest true program impact that would have an 80 percent chance of being 
detected (have 80 percent power) using a two-tailed hypothesis test at the 5 percent level of statistical 
significance. 
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Appendix Table A.3 (continued) 


Outcome 

Number of 

Schools 

MDE 

MDES 

Panel C: High schools 

Attendance 

Percentage of enrolled days attended 

29 

3.12 

0.22 

Percentage of students who attended over 

90 percent of enrolled days 

29 

6.74 

0.15 

Percentage of students who attended over 

85 percent of enrolled days 

29 

4.28 

0.11 

Behavior 

Percentage of enrolled days suspended or expelled 

29 

0.81 

0.15 

Percentage of students who were ever suspended 
or expelled during year 

29 

7.84 

0.21 

Percentage of students who were suspended or 
expelled for 3 or more days 

29 

7.58 

0.24 

Course performance 3 

Percentage of core courses passed 

29 

2.24 

0.08 

Percentage of students who had no core course 
failures during year 

29 

4.38 

0.09 

Percentage of students who had no math course 
failures during year 

29 

9.60 

0.22 

Percentage of students who had no English/ 
language arts course failures during year 

29 

6.05 

0.15 

ABC composite measure b 

Percentage of students above stability threshold 

29 

5.42 

0.11 

Percentage of students with no early warning 
indicators 

29 

5.85 

0.12 


(continued) 


the Diplomas Now model on the percentage of enrolled days attended of 1 .76 percentage points. 
This suggests that if DN schools had an attendance rate at least 1.76 percentage points higher 
than the non-DN schools’ attendance rate, the impact would be statistically significant. Since 
only half the schools are middle schools, the DN schools would need to show an impact of at 
least 2.55 percentage points in the middle school analyses (see Panel B) for the outcome to be 
statistically significant. These MDEs and MDESs represent the best estimation, but they are 
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Appendix Table A.3 (continued) 


SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: The minimum detectable effect (MDE) and minimum detectable effect size (MDES) in 
this table are calculated based on the standard error of the impact estimate (adjusted for random 
assignment blocks, school-level baseline measure of the outcome, and student baseline 
characteristics) and the number of schools in the sample. A statistical significance level of 10 
percent is assumed. The MDES is calculated by dividing the MDE by the standard deviation of the 
outcome measure for non-DN schools. 

Even though the MDE for a given outcome is x, an estimated impact smaller than x can still be 
found to be statistically significant. This is because the calculation of the MDE incorporates not 
only the probability of making a Type 1 error (that is, concluding that there is an impact when in 
fact there is not) but also the probability of making a Type II error (that is, concluding that there is 
no impact when in fact the program was effective). 

a Core courses include math. English/language arts, science, or social studies courses. 
b The “above stability threshold’’ measure is 1 if a student attended over 90 percent of days 
enrolled in the district, was never suspended or expelled, and did not fail any core courses attempted 
during the school year. The "no early warning indicators” measure is 1 if a student attended over 85 
percent of days enrolled in the district, was suspended or expelled for fewer than three days, and did 
not fail any math or English/language arts courses during the school year. If a student has missing 
course data but meets the other two criteria for an ABC composite measure, the student is assigned 
a value of 1 . 

approximate, and it is possible that an impact smaller than the one listed in this table could be 
found to be statistically significant. 5 


Measure Creation and Factor Analysis of Early School and 
Student Outcomes 

The tables in Chapter 3 compare the early school and student outcomes between DN and non- 
DN schools. These exhibits are based on surveys of teachers, administrators, and students. In 
some cases, single survey items were used in the analyses, but in other cases, survey items were 
combined to create a single measure. The following discussion describes the measure creation 
and factor analyses for measures that were constructed out of more than one survey item. 

Items from Table 3.1 

Under “more positive school climate,” Table 3.1 includes the teacher survey measure: 
“Teachers reported rules for student behavior were consistently reinforced by administrators 
and other teachers.” The value for the construct was calculated by taking an average of the 


5 A smaller impact could be statistically significant because the calculation of the MDE incorporates not 
only the probability of making a Type 1 error (that is, concluding that there is an impact when in fact there is 
not) but also the probability of making a Type II error (that is, concluding that there is no impact when in 
fact the program was effective). 
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following survey item responses and then rescaling to a 0-10 scale. (Two items, Cronbach’s 
alpha = 0.79.) 

To the best of your ability, please indicate how frequently the following activities 
occurred at your school during the [current] school year. 

_ Rules for student behavior were consistently reinforced by administra- 
tors. 

_ Rules for student behavior were consistently reinforced by other 
teachers. 

(Scale: 1 = Never, 2 = Rarely, 3 = Sometimes, 4 = Often, 5 = Always, as needed) 

Under “increased parent and community involvement,” Table 3.1 includes the teacher survey 
measure: “Teachers reported their school had effective communication with and participation 
from students’ families and the local community.” The value for the construct was calculated by 
taking an average of the following survey item responses and then rescaling to a 0-10 scale. 6 
Responses that stated “Not offered at the school this year” were coded as 0. (Four items, 
Cronbach’s alpha = 0.91.) 

For the [current] school year, how would you rate your school on each of the 
following? 

_ Having effective communications from school-to-home and from home- 
to-school about school programs and student progress. 

_ Providing a wide range of volunteer opportunities for parents and com- 
munity members to support the school. 

_ Involving families with their children on homework and other curricu- 
lum-related activities and decisions, such as academic goals or plans for 
work or college. 

_ Coordinating resources and services both from the community and to the 
community. 

(Scale: 0 = Not offered at the school this year, 1 = Poor, 2 = Fair, 3 = Good, 4 = 
Excellent) 

Table 3.1 also includes the teacher survey measure: “Teachers reported parents were involved 
in school activities and supported school reform efforts.” The value for the construct was 


6 These survey items come from Markow and Pieters (2012). 
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calculated by taking an average of the following survey item responses and then rescaling to a 
0-10 scale. (Two items, Cronbach’s alpha = 0.83.) 

To what extent would you disagree or agree with each of the following statements 
about your experiences at this school during the [current] school year? 

_ Parents/guardians were involved in school activities (e.g., health fairs, 
fundraisers, sporting events). 

_ Parents/guardians supported school reform efforts, both those planned 
and implemented (e.g., scheduling models, curriculum selection). 

(Scale: 1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly agree) 

Table 3.1 also includes the teacher survey measure: “Teachers reported community members 
were involved in school activities and supported school refomi efforts.” The value for the 
construct was calculated by taking an average of the following survey item responses and then 
rescaling to a 0-10 scale. (Two items, Cronbach’s alpha = 0.83.) 

To what extent would you disagree or agree with each of the following statements 
about your experiences at this school during the [current] school year? 

_ Community Members were involved in school activities (e.g., health 
fairs, fundraisers, sporting events). 

_ Community Members supported school reform efforts, both those 
planned and implemented (e.g., scheduling models, curriculum selec- 
tion). 

(Scale: 1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly agree) 

Table 3.1 includes the administrator survey measure: “Administrators reported parents were 
involved in school activities and supported school refomi efforts.” The value for the construct 
was calculated by taking an average of the following survey item responses and then rescaling 
to a 0-10 scale. (Two items, Cronbach’s alpha = 0.79.) 

To what extent would you disagree or agree with each of the following statements 
about your school during the [current] school year? 

_ Parents/guardians were involved in school activities (e.g., health fairs, 
fundraisers, sporting events). 


72 



_ Parents/guardians supported school reform efforts, both those planned 
and implemented (e.g., scheduling models, curriculum selection). 

(Scale: 1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly agree) 

Table 3.1 also includes the administrator survey measure: “Administrators reported community 
members were involved in school activities and supported school reform efforts.” The value for 
the construct was calculated by taking an average of the following survey item responses and 
then rescaling to a 0-10 scale. (Two items, Cronbach’s alpha = 0.80.) 

To what extent would you disagree or agree with each of the following statements 
about your school during the [current] school year? 

_ Community Members were involved in school activities (e.g., health 
fairs, fundraisers, sporting events). 

_ Community Members supported school reform efforts, both those 
planned and implemented (e.g., scheduling models, curriculum selec- 
tion). 

(Scale: 1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly agree) 

Items from Table 3.2 

Under “more positive school climate,” Table 3.2 includes the student survey measure: 
“The school had problems with bullying/cyber bullying, students fighting, students cutting 
class, and teachers not being able to control the classroom.” The value for the construct was 
calculated by taking an average of the following survey item responses. (Four items, 
Cronbach’s alpha = 0.78.) 

How much of a problem are the following. . . 

_ Bullying/cyber bullying (students repeatedly teasing other students in 
person or online) 

_ Students fighting 

_ Students cutting classes 

_ Teachers not being able to control the classroom 

(Scale: 0 = Not a problem, 1 = A small problem, 2 = A medium problem, 3 = A 
big problem) 
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Table 3.2 also includes the student survey measure: “The school had problems with students 
physically or verbally abusing teachers or other adults.” The value for the construct was 
calculated by taking an average of the following survey item responses. (Two items, Cronbach’s 
alpha = 0.81.) 

How much of a problem are the following. . . 

_ Physical abuse of teachers or other adults in the school by students 
_ Verbal abuse of teachers or other adults in the school by students 

(Scale: 0 = Not a problem, 1 = A small problem, 2 = A medium problem, 3 = A 

big problem) 

Table 3.2 also includes the student survey measure: “The school had problems with destruction 
of school property, students bringing weapons to school, students abusing drugs or alcohol, and 
gang activity.” The value for the construct was calculated by taking an average of the following 
survey item responses. (Four items, Cronbach’s alpha = 0.86.) 

How much of a problem are the following. . . 

_ Vandalism/destruction of school property 
_ Students bringing weapons like knives and guns to school 
_ Students using/abusing drugs and alcohol in school 
_ Students involved in gangs or gangs being on school property 

(Scale: 0 = Not a problem, 1 = A small problem, 2 = A medium problem, 3 = A 

big problem) 

Items from Table 3.3 

Under “greater confidence and self-worth,” Table 3.3 includes the student survey meas- 
ure: “Students believed if they tried hard, did not give up, and had enough time, they could do 
their schoolwork well.” The value for the construct was calculated by taking an average of the 
following survey item responses. (Three items, Cronbach’s alpha = 0.75.) 

How strongly do you agree or disagree . . . 

_ If I try hard, I believe I can do my schoolwork well. 

_ If I don’t give up, I believe I can do schoolwork that is hard. 

_ If I have enough time, I believe I can do well in my schoolwork. 

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 
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Table 3.3 also includes the student survey measure: “Students worried about projects, tests, poor 
grades, and schoolwork in general.” The value for the construct was calculated by taking an 
average of the following survey item responses. (Four items, Cronbach’s alpha = 0.65.) 


How strongly do you agree or disagree .... 

_ When I have a project to do, I worry about it a lot. 

_ I worry about getting bad grades on tests and projects. 

_ When I take tests, I don’t feel very good. 

_ I worry about school and schoolwork. 

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Under “increased engagement and commitment,” Table 3.3 includes the student survey meas- 
ure: “Students thought their work was interesting and liked what they learned in class.” The 
value for the construct was calculated by taking an average of the following survey item 
responses. (Two items, Cronbach’s alpha = 0.79.) 

How often do you agree with the following statements about yourself? 

_ I think that what we are learning in my classes is interesting. 

_ I like what I am learning in my classes. 

(Scale: 0 = Never, 1 = Sometimes, 2 = Very often, 3 = All the time) 

Table 3.3 also includes the student survey measure: “Students tried less at school over time, 
not caring about school, and not participating in class.” The value for the construct was calcu- 
lated by taking an average of the following survey item responses. (Four items, Cronbach’s 
alpha = 0.79.) 

How strongly do you agree or disagree . . . 

_ Each week I try less and less at school. 

_ I don’t really care about school. 

_ I’m not involved in things like class activities and class discussions at 
school. 

_ Fve given up on being interested in school. 

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 
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Table 3.3 also includes the student survey measure: “Students believed learning and working 
hard in class was important for their future.” The value for the construct was calculated by 
taking an average of the following survey item responses. (Five items, Cronbach’s alpha = 
0.83.) 


How strongly do you agree or disagree . . . 

_ My classes prepare me for what I plan to do in life. 

_ What I leam in my classes is important for my future. 

_ Working hard in my classes will matter for my future success in a career. 

_ Learning at school is important to me. 

_ It’s important to me to understand what I’m taught at school. 

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Under “increased effort and persistence,” Table 3.3 includes the student survey measure: 
“Students did not give up easily when they did not understand or when homework was diffi- 
cult.” The value for the construct was calculated by taking an average of the following survey 
item responses. (Four items, Cronbach’s alpha = 0.74.) 

How strongly do you agree or disagree .... 

_ I don’t give up easily. 

_ If I don’t understand my schoolwork, I keep trying until I do. 

_ If my homework is difficult I keep trying to work on it trying to figure it 
out. 

_ When I am taught something that doesn’t make sense, I spend time try- 
ing to understand it. 

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Table 3.3 also includes the student survey measure: “Students paid attention, stayed on task in 
class, and completed all their schoolwork.” The value for the construct was calculated by taking 
an average of the following survey item responses. (Three items, Cronbach’s alpha = 0.80.) 

How often do you agree with the following statements about yourself? 


I pay attention in my classes. 

I stay on task in my classes. 

I complete all my schoolwork. 
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(Scale: 0 = Never, 1 = Sometimes, 2 = Very often, 3 = All the time) 


Under “improved study habits and strategies,” Table 3.3 includes the student survey measure: 
“Students used study strategies such as note taking, graphic organizers, and formulas to help 
remember information.” The value for the construct was calculated by taking an average of the 
following survey item responses. (Four items, Cronbach’s alpha = 0.71.) 

How often do you use the following skills THIS YEAR. . . 

_ Note taking 

_ Mind mapping/graphic organization of information (for example, KWL 
charts and Venn diagrams) 

_ Formulas or rhymes used to help you remember things (mnemonic 
devices) 

_ T est preparation/ test taking skills 
(Scale: 0 = Never, 1 = Sometimes, 2 = Very often, 3 = All the time) 

Table 3.3 also includes the student survey measure: “Students used social skills and conflict 
resolution strategies for controlling anger.” The value for the construct was calculated by taking 
an average of the following survey item responses. (Two items, Cronbach’s alpha = 0.63.) 

How often do you use the following skills THIS YEAR. . . 

_ Conflict resolution/strategies for controlling anger 
_ Social skills (cooperation, active listening, compromise, etc.) 

(Scale: 0 = Never, 1 = Sometimes, 2 = Very often, 3 = All the time) 

Under “positive relationships with adults and peers,” Table 3.3 includes the student survey 
measure: “Students had positive relationships with teachers.” The value for the construct was 
calculated by taking an average of the following survey item responses. (Seven items, 
Cronbach’s alpha = 0.86.) 

How strongly do you agree or disagree . . . 

_ Students at my school get along well with teachers. 

_ My teachers meet with me to talk about schoolwork and give me extra 
help if I need it. 

_ My teachers really listen to what I have to say. 
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_ Teachers at this school set a positive example for students with their ac- 
tions. 

_ My teachers notice when I am doing a good job and let me know about it. 

_ Teachers at my school try to be fair. 

_ If I had a problem outside of the classroom, I felt I could talk to a teacher 
at my school, (high school students only) 

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Table 3.3 also includes the student survey measure: “Administrators were respectful of students 
and teachers and were fair when enforcing rules.” The value for the construct was calculated by 
taking an average of the following survey item responses. (Three items, Cronbach’s alpha = 
0.75.) 


How strongly do you agree or disagree . . . 

_ Principals, Assistant Principals, and other adults in this school are re- 
spectful of students. 

_ Principals, Assistant Principals, and other adults in this school are re- 
spectful of teachers. 

_ Principals, Assistant Principals, and other adults in my school are fair 
when enforcing rules. 

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Table 3.3 also includes the student survey measure: “Students had a positive relationship with at 
least one adult in the school other than a teacher.” The value for the construct was calculated by 
taking an average of the following survey item responses. (Four items, Cronbach’s alpha = 
0.79.) 


How strongly do you agree or disagree . . . 

_ At least one adult at this school other than my teacher(s) really cares 
about me. 

_ At least one adult at this school other than my teacher(s) checks in with 
me almost every day. 

_ At least one adult at this school other than my teacher(s) encourages me 
to do my best. 

_ I could go to at least one adult at this school other than a teacher to get 
help with solving problems at home or school. 
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(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 


Table 3.3 also includes the student survey measure: “Students felt they fit in at school and that 
other students at school accepted them for who they were.” The value for the construct was 
calculated by taking an average of the following survey item responses. (Two items, Cronbach’s 
alpha = 0.68.) 

How strongly do you agree or disagree . . . 

_ Overall, other students at this school accept me for who I am. 

_ I feel that I “fit in” at this school. 

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 
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Appendix B 

Supplemental ABC Outcome Findings 




This appendix offers additional analyses related to the findings discussed in Chapter 4. It 
includes impacts on the attendance, behavior, and course perfonnance (ABC) outcomes for 
subgroups of middle school students who were and were not proficient before participating in 
the study, for the sample of students who attended the study school for the entire implementa- 
tion year, and for Cohort 1 middle school and high school students separately. It also includes 
comparisons of the impacts on the percentage of students with stability indicators and the 
percentage of students with no early warning indicators by random assigmnent block for the 
first and second cohorts of students. Finally, it includes the findings from a sensitivity test of the 
analysis at the middle school level controlling for differences in grade configuration across the 
middle schools. The results of these subgroup and sensitivity analyses help to explain whether 
the main findings presented in Chapter 4 are sensitive to changes in the analytic sample or to 
differences in school grade configuration, and whether the growth in impacts from the first to 
the second cohort vary across the schools in the study. 


Impacts for Middle School Students by Prior-Year Proficiency 

Appendix Table B.l presents the ABC outcomes for subgroups of middle school students 
designated by whether they scored at or above proficiency on math and English state standard- 
ized tests in the previous year. The middle school students in Panel A received a proficient 
rating on both their English/language arts and math assessments in the previous school year. 
Panel B presents the findings for middle school students who were not proficient on one or both 
assessments in the prior year. Some students were not included in these analyses because their 
test scores for the previous school year were not available. Twenty-eight percent, or 1,261, of 
the students are in the proficient subgroup, while 72 percent, or 3,185 students, are in the 
nonproficient subgroup. 

In general, findings for the proficient and nonproficient subgroups are similar, and no sta- 
tistically significant differences between these two groups were found, suggesting that the 
Diplomas Now model supported both these groups to a similar degree. There were two statistical- 
ly significant impacts for the nonproficient subgroup. First, the impact of Diplomas Now on the 
proportion of students with at least 90 percent attendance is positive. While the impact on this 
measure for the students who were proficient on both tests in the previous year is not statistically 
significant, the effect size of this outcome is larger for the proficient students than for the non- 
proficient students, and the difference between these two impacts is not statistically significant. 1 


'Since there are fewer students in the proficient subgroup, the model for that subgroup has less statistical 
power in comparison with the nonproficient subgroup and cannot detect statistically significant impacts at the 
same level. 
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Appendix Table B.l 


Impacts on Threshold Measures of 
Attendance, Behavior, and Course Performance, 

Prior-Year Math and English Proficiency Subgroups, Middle Schools, Cohort 2 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P-Value for 

Estimated 

Impact 

Panel A: Proficient on both tests 

Attendance 

Percentage of students who attended over 

90 percent of enrolled days 

82.3 

77.6 

4.7 

0.14 

0.213 

Percentage of students who attended over 

85 percent of enrolled days 

91.2 

87.2 

4.0 

0.14 

0.426 

Behavior 

Percentage of students who were ever suspended 
or expelled during year 

15.9 

16.9 

-0.9 

-0.02 

0.873 

Percentage of students who were suspended or 
expelled for 3 or more days 

11.8 

10.1 

1.7 

0.05 

0.698 

Course performance 

Percentage of students who had no core course 
failures during year 3 

93.8 

93.9 

-0.1 

-0.01 

0.959 

Percentage of students who had no math course 
failures during year 

95.2 

96.3 

-1.0 

-0.08 

0.520 

Percentage of students who had no English/ 
language arts course failures during year b 

99.2 

97.6 

1.6 

0.12 

0.294 

ABC composite measure 0 

Percentage of students above stability threshold 

72.8 

64.7 

8.1 

0.17 

0.145 

Percentage of students with no early warning 
indicators 

80.9 

76.6 

4.3 

0.10 

0.423 

Sample size 

14 

15 





(continued) 


Second, there was a statistically significant impact on the percentage of students who had no early 
warning indicators in the nonproficient subgroup. The proficient subgroup also had a positive 
though not statistically significant impact on that outcome, and again the differences between the 
two findings are not statistically significant. 
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Appendix Table B.l (continued) 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P- Value for 

Estimated 

Impact 

Panel B: Not proficient on one or both tests 

Attendance 

Percentage of students who attended over 

90 percent of enrolled days 

77.1 

72.5 

4.6 

0.10 * 

0.069 

Percentage of students who attended over 

85 percent of enrolled days 

87.3 

84.2 

3.1 

0.08 

0.234 

Behavior 

Percentage of students who were ever suspended 
or expelled during year 

25.8 

25.4 

0.4 

0.01 

0.947 

Percentage of students who were suspended or 
expelled for 3 or more days 

18.6 

19.6 

-1.0 

-0.03 

0.821 

Course performance 

Percentage of students who had no core course 
failures during year' 1 

83.9 

82.7 

1.2 

0.03 

0.744 

Percentage of students who had no math course 
failures during year 

92.7 

93.1 

-0.4 

-0.01 

0.869 

Percentage of students who had no English/ 
language arts course failures during year 

94.7 

92.6 

2.2 

0.09 

0.217 

ABC composite measure 0 

Percentage of students above stability threshold 

59.5 

54.1 

5.4 

0.11 

0.182 

Percentage of students with no early warning 

indicators 

72.2 

65.2 

7.0 

0.15 ** 

0.036 

Sample size 

14 

15 





(continued) 


Impacts for the Full-Year Sample of Students 

In general, students who attended the study school for the full duration of the school year 
represent a “stable sample” of students. More specifically, the criteria for including students 
in the full-year sample subgroup are as follows: (1) the student entered the study school 
within 14 days of the first day of school, (2) the student did not exit the study school more 
than 14 days before the end of the school year, (3) the student did not have a lapse in attend- 
ance of greater than 14 days, and (4) the student did not attend both a Diplomas Now (DN) 
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Appendix Table B.l (continued) 

SOURCE: MDRC calculations based on student records, inclusive of state assessment data, obtained from school 
districts. 

NOTES: Across 29 study middle schools, 4,446 nonrepeating sixth-grade students are included in the analyses. 
Among the sample, 2,030 students attended DN schools and 2,416 students attended non-DN schools. The 
proficient subgroup includes 1,261 students and the nonproficient subgroup includes 3,185 students. Some students 
are not included in the analyses of course performance measures because data were not available on their grades for 
specific courses. An entire DN middle school was dropped from these analyses because there are no baseline 
course data for the students attending that school. There are no more than 8 percent missing DN school students 
and 14 percent missing non-DN school students for any of the course performance measures. 

Estimated impacts are based on a two-level model with students nested within schools, controlling for random 
assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 
observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 
column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 
covariate values for students in DN schools as the basis for the adjustment. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistically significant differences in impacts between the proficient and nonproficient groups are indicated by f 
when the p-value is less than or equal to 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Core courses include math. English/language arts, science, or social studies courses. 

b One random assignment block, which includes one DN school and one non-DN school, was removed from this 
analysis because lack of variation in that random assignment block was impeding the analysis from running 
properly. 

c The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 
district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 
"no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, was 
suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 
the school year. If a student has missing course data but meets the other two criteria for an ABC composite 
measure, the student is assigned a value of 1 . 


and a non-Diplomas Now (non-DN) study school during the school year. The full-year 
sample subgroup comprises 10,948 students, which is approximately 73 percent of the full 
study sample. Appendix Table B.2 presents the impact results of the full-year sample sub- 
group analysis for Cohort 2. Similar to the main impact findings presented in Table 4.2, the 
full-year sample subgroup analyses yielded one statistically significant impact of the Diplo- 
mas Now model on the percentage of students with no early warning indicators. 
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Appendix Table B.2 


Impacts on Continuous and Threshold Measures of 
Attendance, Behavior, and Course Performance, Full-Year Sample, Cohort 2 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P- Value for 

Estimated 

Impact 

Attendance 

Percentage of enrolled days attended 

91.8 

91.7 

0.1 

0.01 

0.863 

Percentage of students who attended over 

90 percent of enrolled days 

73.8 

74.0 

-0.3 

-0.01 

0.877 

Percentage of students who attended over 

85 percent of enrolled days 

84.9 

85.0 

-0.1 

0.00 

0.963 

Behavior 

Percentage of enrolled days suspended or expelled 

1.1 

0.8 

0.3 

0.11 

0.161 

Percentage of students who were ever suspended 
or expelled during year 

23.8 

22.0 

1.7 

0.04 

0.631 

Percentage of students who were suspended or 
expelled for 3 or more days 

16.2 

14.8 

1.4 

0.04 

0.633 

Course performance 2 

Percentage of core courses passed 

88.9 

88.9 

0.0 

0.00 

0.977 

Percentage of students who had no core course 
failures during year 

76.7 

76.4 

0.3 

0.01 

0.873 

Percentage of students who had no math course 
failures during year 

86.8 

85.1 

1.8 

0.05 

0.430 

Percentage of students who had no English/ 
language arts course failures during year 

90.1 

88.8 

1.2 

0.04 

0.357 

ABC composite measure b 

Percentage of students above stability threshold 

54.4 

52.3 

2.0 

0.04 

0.334 

Percentage of students with no early warning 

indicators 

67.7 

63.7 

4.0 

0.08 ** 

0.033 

Sample size 

29 

29 





(continued) 
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Appendix Table B.2 (continued) 


SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Students are included in the full-year sample if they meet the following criteria: (1) entered the study 
school within 14 days of the first day of school, (2) did not exit the study school more than 14 days before the end 
of the school year, (3) did not have a lapse of attendance data greater than 14 days, and (4) did not attend both a 
DN and non-DN study school during the school year. 

Across 58 study schools, 10,948 nonrepeating sixth- and ninth-grade students are included in the full-year 
sample and analyses. Among the sample, 5,069 students attended DN schools and 5,879 students attended non-DN 
schools. Some students are not included in the analyses of course performance measures because data were not 
available on their grades for specific courses. An entire DN middle school was dropped from these analyses 
because there are no baseline course data for the students attending that school. There are no more than 7 percent 
missing DN school students and 2 percent missing non-DN school students for any of the course performance 
measures. The large percentage of missing DN school students is due to one school with a large amount of missing 
math course data. Without this school included, there are no more than 2 percent missing students on any of the 
measures. A sensitivity test was done removing the random assignment block that includes this school from the 
analyses and findings were similar. 

Estimated impacts are based on a two-level model with students nested within schools, controlling for random 
assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 
observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 
column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 
covariate values for students in DN schools as the basis for the adjustment. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Core courses include all math. English/language arts, science, or social studies courses. 

b The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 
district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 
“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, was 
suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 
the school year. If a student has missing course data but meets the other two criteria for an ABC composite 
measure, the student is assigned a value of 1 . 


Impacts for Cohort 1 Middle School and High School Subgroups 

Appendix Table B.3 presents the impacts of the Diplomas Now model for the sixth- and ninth- 
grade students who attended the study schools during the first year of implementation (Cohort 
1), separated into middle school and high school subgroups. These subgroup analyses yielded 
only one statistically significant impact of the Diplomas Now model, on the percentage of sixth- 
grade students who had no math course failures. This is consistent with the findings presented in 
Table 4.6 for the full Cohort 1 study sample (middle and high schools combined), where there 
was a statistically significant impact on this same measure and no others. 



Appendix Table B.3 


Impacts on Threshold Measures of Attendance, Behavior, 
and Course Performance, Middle Schools and High Schools, Cohort 1 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated Effect 

Impact Size 

P- Value for 

Estimated 

Impact 

Panel A: Middle schools 

Attendance 

Percentage of students who attended over 

90 percent of enrolled days 

76.1 

76.5 

-0.4 -0.01 

0.871 

Percentage of students who attended over 

85 percent of enrolled days 

86.7 

86.9 

-0.2 -0.01 

0.837 

Behavior 

Percentage of students who were ever suspended 
or expelled during year 

23.0 

19.1 

3.9 0.10 

0.490 

Percentage of students who were suspended or 
expelled for 3 or more days 

15.4 

12.7 

2.8 0.08 

0.518 

Course performance 

Percentage of students who had no core course 
failures during year a 

88.5 

84.8 

3.7 0.10 

0.320 

Percentage of students who had no math course 
failures during year 

96.5 

91.2 

5.3 0.18 ** 

0.029 

Percentage of students who had no English/ 
language arts course failures during year 

95.2 

94.2 

1.0 0.04 

0.629 

ABC composite measure b 

Percentage of students above stability threshold 

59.4 

62.9 

-3.6 -0.07 

0.306 f 

Percentage of students with no early warning 

indicators 

73.2 

73.7 

-0.6 -0.01 

0.859 

Sample size 

16 

15 




(continued) 


There was a statistically significant difference between the impact estimates for Cohort 
1 and Cohort 2 middle school students on the percentage of students with stability indicators, as 
represented by the dagger on the far right side of the table. Although not statistically significant 
for either cohort, the impact estimate was negative for Cohort 1 and positive for Cohort 2. The 
dagger suggests the Diplomas Now model grew more successful in supporting middle school 
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Appendix Table B.3 (continued) 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P-Value for 

Estimated 

Impact 

Panel B: High schools 

Attendance 

Percentage of students who attended over 

90 percent of enrolled days 

56.9 

58.9 

-2.0 

-0.04 

0.471 

Percentage of students who attended over 

85 percent of enrolled days 

69.0 

70.2 

-1.1 

-0.03 

0.768 

Behavior 

Percentage of students who were ever suspended 
or expelled during year 

28.5 

25.7 

2.8 

0.07 

0.134 

Percentage of students who were suspended or 
expelled for 3 or more days 

20.9 

20.2 

0.7 

0.02 

0.798 

Course performance 

Percentage of students who had no core course 
failures during year 3 

57.9 

55.1 

2.8 

0.06 

0.351 

Percentage of students who had no math course 
failures during year 

75.6 

73.2 

2.3 

0.05 

0.448 

Percentage of students who had no English/ 
language arts course failures during year 

77.6 

76.4 

1.2 

0.03 

0.727 

ABC composite measure b 

Percentage of students above stability threshold 

34.4 

35.5 

-1.1 

-0.02 

0.506 

Percentage of students with no early warning 

indicators 

47.9 

46.6 

1.3 

0.03 

0.560 

Sample size 

15 

14 





(continued) 


students to get or stay on a stable path to graduation between the first and second years of 
implementation. 


Comparison of Random Assignment Block-Level Impacts 
by Cohort 

Appendix Tables B.4 and B.5 present random assignment block-level impact estimates for the 
percentage of students who had stability indicators and the percentage of students who had no 
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Appendix Table B.3 (continued) 


SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 60 study schools, 5,729 nonrepeating sixth-grade students and 9,551 nonrepeating ninth-grade 
students are included in the analyses. Among the middle school sample, 2,671 students attended DN schools and 
3,058 students attended non-DN schools. Among the high school sample, 4,651 students attended DN schools and 
4,900 students attended non-DN schools. Some students are not included in the analyses of course performance 
measures because data were not available on their grades for specific courses. An entire DN middle school was 
dropped from these analyses because there are no baseline course data for the students attending that school. There 
are no more than 10 percent missing DN school students and 14 percent missing non-DN school students for any of 
the course performance measures. 

Estimated impacts are based on a two-level model with students nested within schools, controlling for random 
assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 
observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 
column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 
covariate values for students in DN schools as the basis for the adjustment. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistically significant differences in impacts between the impact estimates for Cohort 1 and Cohort 2 are 
indicated by f when the p-value is less than or equal to 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Core courses include math. English/language arts, science, or social studies courses. 

b The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 
district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 
"no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, was 
suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 
the school year. If a student has missing course data but meets the other two criteria for an ABC composite 
measure, the student is assigned a value of 1 . 


early warning indicators, respectively. The first set of columns in each of these tables lists the 
impacts for each random assignment block for the first cohort of sixth- and ninth-grade stu- 
dents, and the second set of columns presents the same information for the second cohort. The 
tables also present the differences between the Cohort 2 and Cohort 1 impacts. Appendix Table 
B.4 shows that 11 of the random assignment blocks had an increase in the impact on the 
percentage of students with stability indicators between Cohort 1 and Cohort 2, while 10 blocks 
experienced a decrease in the impact by the second year. The table also shows that in two 
random assignment blocks, there were statistically significant positive differences in impacts for 
Cohort 2 compared with Cohort 1. Appendix Table B.5 shows that 12 of the random assign- 
ment blocks experienced an increase in the impact on the percentage of students with no early 
warning indicators between Cohort 1 and Cohort 2, while 9 blocks experienced a decrease in 
the impact. One random assignment block had a statistically significant positive difference in 
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Appendix Table B.4 


Comparison of Cohort 1 and Cohort 2 Impacts 
by Random Assignment Block, 
Percentage of Students Above Stability Threshold 


Random 

Assignment 

Block 

Cohort 1 

Cohort 2 

Difference 

Impact 

P- Value 

Impact 

P-Value 

(C2-C1) 

P- Value 

109 

-20.2 * 

0.052 

6.6 

0.527 

26.9 * 

0.066 

112 

-17 8 *** 

0.003 

8.4 

0.171 

26.2 *** 

0.002 

118 

-1.6 

0.876 

21.3 * 

0.059 

22.9 

0.129 

101 

0.4 

0.964 

15.4 * 

0.097 

15.0 

0.214 

107 

-13.8 

0.136 

-1.9 

0.847 

11.9 

0.363 

108 

-15.0 

0.173 

-3.6 

0.752 

11.4 

0.464 

104 

-14.9 

0.248 

-6.7 

0.624 

8.2 

0.658 

122 

-5.6 

0.370 

0.7 

0.915 

6.3 

0.486 

120 

-1.5 

0.867 

4.8 

0.615 

6.2 

0.626 

113 

8.6 

0.343 

13.7 

0.173 

5.2 

0.695 

119 

1.1 

0.889 

2.3 

0.786 

1.2 

0.916 

115 

-4.5 

0.476 

-5.5 

0.422 

-1.0 

0.914 

111 

10.1 

0.290 

9.1 

0.376 

-1.0 

0.941 

102 

-9.0 

0.309 

-11.1 

0.275 

-2.1 

0.870 

117 

-11.0 

0.251 

-14.3 

0.174 

-3.3 

0.810 

116 

12.0 

0.197 

7.1 

0.476 

-5.0 

0.708 

103 

8.8 

0.258 

3.0 

0.716 

-5.8 

0.602 

106 

13.2 

0.180 

6.9 

0.504 

-6.2 

0.656 

110 

5.2 

0.565 

-5.3 

0.586 

-10.4 

0.425 

105 

29.2 ** 

0.012 

17.9 

0.116 

-11.3 

0.455 

114 

13.4 * 

0.052 

-1.3 

0.856 

-14.7 

0.130 

Sample size 

60 


58 





SOURCE: MDRC calculations based on student records obtained from school districts. 


NOTES: Across 60 study schools in Cohort 1 and 58 study schools in Cohort 2, 15,280 Cohort 1 
students and 14,950 Cohort 2 students are included in the analyses. 

The “above stability threshold’’ measure is 1 if a student attended over 90 percent of days enrolled 
in the district, was never suspended or expelled, and did not fail any core courses attempted during the 
school year. If a student has missing course data but meets the other two criteria for an ABC composite 
measure, the student is assigned a value of 1 . 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 
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Appendix Table B.5 


Comparison of Cohort 1 and Cohort 2 Impacts 
by Random Assignment Block, 

Percentage of Students with No Early Warning Indicators 


Random 

Assignment 

Block 

Cohort 1 

Cohort 2 

Difference 

Impact 

P-Value 

Impact 

P-Value 

(C2-C1) 

P-Value 

118 

-4.4 

0.664 

16.4 * 

0.085 

20.7 

0.129 

109 

-6.1 

0.524 

13.6 

0.137 

19.7 

0.133 

112 

-7.8 

0.128 

9.6 * 

0.063 

17.3 ** 

0.016 

104 

-25.0 ** 

0.033 

oo 

O O 

i 

0.408 

16.2 

0.287 

108 

-5.7 

0.573 

7.4 

0.426 

13.1 

0.336 

107 

-11.5 

0.192 

0.2 

0.980 

11.7 

0.312 

101 

2.6 

0.739 

13.5 * 

0.075 

10.9 

0.304 

113 

9.9 

0.261 

12.3 

0.140 

2.3 

0.842 

102 

-7.0 

0.418 

-5.5 

0.535 

1.6 

0.898 

119 

1.9 

0.802 

3.3 

0.639 

1.4 

0.894 

122 

0.4 

0.952 

1.5 

0.783 

1.1 

0.887 

120 

6.3 

0.457 

6.8 

0.378 

0.5 

0.963 

117 

-5.7 

0.530 

-6.4 

0.446 

-0.7 

0.953 

111 

10.7 

0.251 

10.0 

0.252 

-0.7 

0.952 

110 

-2.9 

0.734 

-4.7 

0.548 

-1.8 

0.877 

103 

7.0 

0.353 

4.3 

0.528 

-2.7 

0.783 

115 

-0.8 

0.894 

-4.3 

0.449 

-3.5 

0.671 

106 

14.0 

0.153 

7.6 

0.396 

-6.4 

0.619 

114 

7.8 

0.230 

0.9 

0.877 

-6.9 

0.423 

116 

14.1 

0.117 

2.7 

0.736 

-11.5 

0.329 

105 

31.3 *** 

0.006 

5.9 

0.528 

-25.4 * 

0.070 

Sample size 

60 


58 





SOURCE: MDRC calculations based on student records obtained from school districts. 


NOTES: Across 60 study schools in Cohort 1 and 58 study schools in Cohort 2, 15,280 Cohort 1 
students and 14,950 Cohort 2 students are included in the analyses. 

The “no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled 
in the district, was suspended or expelled for fewer than three days, and did not fail any math or 
English/language arts courses during the school year. If a student has missing course data but meets the 
other two criteria for an ABC composite measure, the student is assigned a value of 1. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 
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the impact for Cohort 2 compared with Cohort 1, and one random assignment block had a 
statistically significant decrease in the impact between the first and second cohorts. 


Middle School Impacts Controlling for Grade Configuration 

Appendix Table B.6 provides the results of a sensitivity analysis that was conducted to verily 
that the estimates of program impacts presented in Chapter 4 of this report are unbiased and that 
they can be interpreted as the effect of the Diplomas Now intervention and are not related to 
other measurable school characteristics. For this sensitivity analysis, an additional school-level 
covariate indicating middle school configuration was added to the analytic model described in 
Appendix A. This decision was made based on prior research suggesting that middle school 
configuration can affect student outcomes and, in particular, that transitioning to a new school 
can have an effect on a student’s outcomes during that year. 2 

The grade configuration covariate has a value of one if the school the student attended 
has a grade configuration of fifth through eighth grade or kindergarten through eighth grade. 
Because sixth grade is not the first grade level offered at these schools, the sixth-grade students 
in the analysis were not generally transitioning to a new school during this year. If the school a 
student attended serves only sixth- through eighth-graders or seventh- and eighth-graders, the 
covariate has a value of zero. 3 Six of the 29 study middle schools (21 percent) received a value 
of one, while the remaining 23 received a value of zero. 

The results of this sensitivity analysis show several statistically significant impacts. 
Similar to Table 4.3, which displays the impacts for the middle school subgroup without the 
added grade configuration covariate, there were positive and statistically significant impacts on 
the percentage of students who had a 90 percent attendance rate or better and on the percentage 
of students who have no early warning indicators at the end of the school year. In both cases the 
analysis including the school configuration covariate results in a lower p-value. The other two 
outcomes that had statistically significant impacts when the model controlled for middle school 
grade configuration are the percentage of core courses passed and the percentage of students 
who had no English/language arts course failures during the year. In both cases these findings 
are positive but not statistically significant in Table 4.1 (Panel B) and Table 4.3, although in the 
case of the percentage of students who had no English/language arts course failures the p-value 
is very near significant at 0. 101 . 


2 Cook, MacCoun, Muschkin, and Vigdor (2008); Rockoff and Lockwood (2010); Schwartz, Stiefel, 
Rubenstein, and Zabel (2011); Schwerdt and West (2013). 

3 ln schools with only seventh- and eighth-graders, seventh-graders were included in the analyses because 
seventh grade was the transition year. 
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Appendix Table B.6 


Impacts on Continuous and Threshold Measures of 
Attendance, Behavior, and Course Performance, 
Accounting for Variation in School Grade Configuration, 
Middle Schools, Cohort 2 


Outcome 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Impact 

Effect 

Size 

P- Value for 

Estimated 

Impact 

Attendance 

Percentage of enrolled days attended 

92.7 

91.1 

1.6 

0.14 

0.161 

Percentage of students who attended over 

90 percent of enrolled days 

77.2 

72.2 

5.0 

o n ** 

0.046 

Percentage of students who attended over 

85 percent of enrolled days 

86.8 

83.8 

3.0 

0.08 

0.285 

Behavior 

Percentage of enrolled days suspended or expelled 

1.2 

1.2 

-0.1 

-0.01 

0.785 

Percentage of students who were ever suspended 
or expelled during year 

22.1 

21.9 

0.2 

0.00 

0.976 

Percentage of students who were suspended or 
expelled for 3 or more days 

15.6 

16.4 

-0.8 

-0.02 

0.864 

Course performance 3 

Percentage of core courses passed 

94.5 

92.4 

2.1 

0.12 * 

0.068 

Percentage of students who had no core course 
failures during year 

87.0 

84.8 

2.2 

0.06 

0.425 

Percentage of students who had no math course 
failures during year 

94.3 

93.6 

0.7 

0.03 

0.757 

Percentage of students who had no English/ 
language arts course failures during year 

95.5 

93.0 

2.5 

0.12 ** 

0.021 

ABC composite measure 13 

Percentage of students above stability threshold 

62.5 

56.5 

5.9 

0.12 

0.123 

Percentage of students with no early warning 
indicators 

74.5 

68.1 

6.4 

0.14 * 

0.059 

Sample size 

14 

15 





(continued) 
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Appendix Table B.6 (continued) 


SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 29 study schools, 5,606 nonrepeating sixth-graders are included in the analyses. Among the 
sample, 2,513 students attended DN schools and 3,093 students attended non-DN schools. Some students are not 
included in the analyses of course performance measures because data were not available on their grades for 
specific courses. An entire DN middle school was dropped from these analyses because there were no baseline 
course data for the students attending that school. There are no more than 9 percent missing DN school students 
and 12 percent missing non-DN school students for any of the course performance measures. 

Estimated impacts are based on a two-level model with students nested within schools, controlling for random 
assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 
observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 
column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 
covariate values for students in DN schools as the basis for the adjustment. 

Variation in school grade configuration was accounted for by including a covariate in the analysis that was 
equal to 1 if the school was kindergarten through eighth grade or fifth through eighth grade. 

Effect sizes were computed using the standard deviations of all non-DN school students for the respective 
measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 
indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Core courses include math. English/language arts, science, or social studies courses. 

b The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 
district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 
“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, was 
suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 
the school year. If a student has missing course data but meets the other two criteria for an ABC composite 
measure, the student is assigned a value of 1 . 


96 



References 


Allensworth, Elaine Marie, and John Q. Easton. 2005. The On-Track Indicator as a Predictor 
of High School Graduation. Chicago: University of Chicago Consortium on Chicago School 
Research. 

Balfanz, Robert, John M. Bridgeland, Joanna Homig Fox, Jennifer L. DePaoli, Erin S. Ingram, and 
Mary Maushard. 2014. Building a Grad Nation: Progress and Challenge in Ending the High 
School Dropout Epidemic. Washington, DC: Civic Enterprises. 

Balfanz, Robert, Liza Herzog, and Douglas J. Mac Iver. 2007. “Preventing Student Disengagement 
and Keeping Students on the Graduation Path in Urban Middle-Grades Schools: Early Identifi- 
cation and Effective Interventions.” Educational Psychologist 42, 4: 223-235. 

Borman, Geoffrey D., Gina M. Hewes, Laura T. Overman, and Shelly Brown. 2003. “Comprehen- 
sive School Reform and Achievement: A Meta-Analysis.” Review of Educational Research 73, 
2: 125-230. 

Cook, Phillip J., Robert MacCoun, Clara Muschkin, and Jacob Vigdor. 2008. “The Negative Im- 
pacts of Starting Middle School in Sixth Grade.” Journal of Policy Analysis and Management 
27, 1: 104-121. 

Corrin, William, Susan Sepanik, Aracelis Gray, Felix Fernandez, Ashley Briggs, and Kathleen K. 
Wang. 2014. Laying Tracks to Graduation: The First Year of Implementing Diplomas Now. 
New York: MDRC. 

Corrin, William, Susan Sepanik, Rachel Rosen, and Andrea Shane. 2016. Survey Instrument Sup- 
plement to Addressing Early Warning Indicators: Interim Impact Findings from the Investing 
in Innovation (i3) Evaluation of Diplomas Now. New York: MDRC. 

Frazelle, Sarah, and Aisling Nagel. 2015. A Practitioner’s Guide to Implementing Early Warning 
Systems (REL 2015-056). Washington, DC: U.S. Department of Education, Institute of Educa- 
tion Sciences, National Center for Education Evaluation and Regional Assistance, Regional 
Educational Laboratory Northwest. 

Kemple, James J., Micha D. Segeritz, and Nickisha Stephenson. 2013. “Building On-Track Indica- 
tors for High School Graduation and College Readiness: Evidence from New York City.” 
Journal of Education for Students Placed at Risk (JESPAR) 18, 1: 7-28. 

Markow, Dana, and Andrea Pieters. 2012. The MetLife Survey of the American Teacher: Teachers, 
Parents, and the Economy. New York: MetLife, Inc. 

Moore, Kristin Anderson. 2014. Making the Grades: Assessing the Evidence for Integrated Student 
Supports. Bethesda, MD: Child Trends. 

Mumane, Richard J. 2013. “U.S. High School Graduation Rates: Patterns and Explanations.” Jour- 
nal of Economic Literature 51, 2: 370-422. 


97 



National Center for Education Statistics. 2014. Web Tables — Profile of Undergraduate Students: 
2011-12 (NCES 2015-167). Washington, DC: National Center for Education Statistics, 

Institute of Education Sciences, U.S. Department of Education. 

Plotts, Chris, and Jennifer Sable. 2010. Characteristics of the 100 Largest Public Elementary and 
Secondary School Districts in the United States: 2007-08 (NCES 2010-349). Washington, DC: 
U.S. Department of Education, National Center for Education Statistics. 

Rockoff, Jonah E., and Benjamin B. Lockwood. 2010. “Stuck in the Middle: Impacts of Grade Con- 
figuration in Public Schools.” Journal of Public Economics 94, 11: 1051-1061. 

Roderick, Melissa, Thomas Kelley-Kemple, David W. Johnson, and Nicole O. Beechum. 2014. 
Preventable Failure: Improvements in Long-Term Outcomes When High Schools Focused 
on the Ninth Grade Year. Chicago: University of Chicago Consortium on Chicago School 
Research. 

Schwartz, Amy Ellen, Leanna Stiefel, Ross Rubenstein, and Jeffrey Zabel. 2011. “The Path Not 
Taken: How Does School Organization Affect Eighth-Grade Achievement?” Educational 
Evaluation and Policy Analysis 33, 3: 293-317. 

Schwerdt, Guido, and Martin R. West. 2013. “The Impact of Alternative Grade Configurations on 
Student Outcomes Through Middle and High School.” Journal of Public Economics 97: 308- 
326. 

Sepanik, Susan, William Corrin, David Roy, Aracelis Gray, Felix Fernandez, Ashley Briggs, and 
Kathleen K. Wang. 2015. Moving Down the Track: Changing School Practices During the 
Second Year of Diplomas Now. New York: MDRC. 


98 



Earlier MDRC Publications on Diplomas Now 


Moving Down the Track 

Changing School Practices During the Second Year of Diplomas Now 

2015. Susan Sepanik, William Corrin, David Roy, Aracelis Gray, Felix Fernandez, 

Ashley Briggs, Kathleen K. Wang 

Laying Tracks to Graduation 

The First Year of Implementing Diplomas Now 

2014. William Corrin, Susan Sepanik, Aracelis Gray, Felix Fernandez, Ashley Briggs, 
Kathleen K. Wang 

Data Collection Instrument Supplement to Laying Tracks to Graduation 
The First Year of Implementing Diplomas Now 

2014. William Corrin, Susan Sepanik, Aracelis Gray, Felix Fernandez, Ashley Briggs, 
Kathleen K. Wang 


NOTE: All the publications listed above are available for free download at www.mdrc.org. 


99 



About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of 
social and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) 
and evaluations of ongoing government and community initiatives. MDRC’s staff bring an 
unusual combination of research and organizational experience to their work, providing 
expertise on the latest in qualitative and quantitative methods and on program design, 
development, implementation, and management. MDRC seeks to learn not just whether a 
program is effective but also how and why the program’s effects occur. In addition, it tries to 
place each project’s findings in the broader context of related research — in order to build 
knowledge about what works across the social and education policy fields. MDRC’s findings, 
lessons, and best practices are proactively shared with a broad audience in the policy and 
practitioner community as well as with the general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy 
areas and target populations. Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local 
governments, public school systems, community organizations, and numerous private 
philanthropies. 


